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ABSTRACT 


Test and Evaluation (T&E) is an integral part of every acquisition program, as 
such, it consumes considerable program resources. The Department of Defense (DoD) 
T&E program management requirements are written to meet the risk reduction needs of 
large acquisition programs, but do not provide the details needed to consistently scale 
T&E management efforts for smaller programs across DoD. This research study 
investigates ways that the T&E burden to programs differs based upon the Acquisition 
Category (ACAT), application of T&E oversight, and commodity area. The primary data 
source for the study was each program’s Test and Evaluation Master Plan (TEMP). The 
majority of the programs evaluated are part of the DoD Chemical Biological Defense 
Program (CBDP). This research effort found ACAT-based patterns in requirements 
definitions and ACAT-induced variability in test site usage. The study also determined 
that the TEMP documentation burden was similar across all ACATs, and consistently 
higher than Service guidance. Future studies using this analysis methodology are 
required to detennine whether the T&E burden to CBDP programs is typical or unique 
within DoD. 
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I. 


INTRODUCTION 


A. BACKGROUND 

The Department of Defense (DoD) and individual Service (Army, Navy, Air 
Force, and Marine Corps) guidance for acquisition system testing focuses primarily on 
large Acquisition Category I (ACAT I) programs. Therefore, managers of medium, 
AC AT II, and small, ACAT III, DoD programs must work closely with the Test and 
Evaluation (T&E) community to effectively tailor both test documentation and test scope 
to balance the test burden with other program requirements. A Program, Project, or 
Product Manager (PM) is required to execute their acquisition program within the cost, 
schedule, and technical constraints of the program baseline. The rest of the acquisition 
community, including the T&E organizations, is not constrained by these same overall 
goals. 

Anecdotal PM Office (PMO) experiences point to historical T&E community 
inflexibility, especially in the tailoring of T&E documentation scope based upon program 
size and complexity. While some programs require specialized testing, such as 
laboratory chemical or biological agent testing, most general types of program tests are 
required across-the-board. While many of these tests are applicable to the majority of 
weapon systems, the scope of T&E and the associated documentation should still be 
tailored to strike a balance between user needs and overall program constraints. PMO 
experiences also point to a proclivity to document and test more, rather than less, 
whenever a choice must be made by the T&E activities. Requiring additional tests or 
more detailed documentation is unlikely to cause any personal or organizational issues 
for a tester, while agreeing to less of each will likely raise thorny questions. 

The general PMO sense that the T&E burden, driven by the technical nature of 
the requirements for smaller systems, standardized T&E documentation without regard 
for program scope, and the lack of program-tailored test efforts were the issues that 
sparked this research study. The three research questions defined below were chosen 

specifically to investigate whether these PMO-held notions regarding T&E are justified. 
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B. RESEARCH QUESTIONS 


1. Primary Research Question 

• Is the T&E Level of Effort (LOE) proportional to a program’s designated 
ACAT level? 

2. Secondary Research Questions 

• How does program oversight by the Director Operational Test and 

Evaluation (DOT&E) or the Army drive the T&E LOE? 

• Are Chemical/Biological (C/B) programs a special case for T&E LOE? 

C. SCOPE AND METHODOLOGY 

1. Scope 

This research study will evaluate the T&E burden for acquisition programs, based 
primarily on the data available in each program’s TEMP. The study focuses on the 
CBDP within DoD, but also includes data for two ACAT I non-CBDP programs. A 
detailed analysis is performed to determine differences in T&E requirements, 
documentation burden, and other related factors that impact the scope of T&E for 
acquisition programs. 

This study is bounded by the following conditions: 

• Only programs with a TEMP containing sufficient detail for a comparative 
analysis were included in this study. 

• Software-based programs, such as Infonnation Technology (IT) and 
Command, Control, Communications, Computers, Intelligence, 
Surveillance and Reconnaissance (C4ISR) were not evaluated. 

• The inconsistent nature of program cost summaries between TEMPs, and 
as reported in DoD budget materials, precluded an analysis of costs, either 
at the program-level or for T&E efforts. 

2. Methodology 

A summary of the approach used to conduct this research project is: 
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• Conduct a literature search, of both the publicly-accessible and the For 
Official Use Only (FOUO) sections, of the Joint Program Executive 
Office for Chemical Biological Defense (JPEO-CBD) program database to 
acquire CBDP-specific acquisition program documents. 

• Gather publicly-available funding data on CBDP programs from the DoD 
budget materials contained on the Office of the Under Secretary of 
Defense, Comptroller (OUSD(C)) web site to supplement program- 
specific acquisition documents. 

• Investigate the availability and usefulness of detailed T&E documents, 
such as Detailed Test Plans (DTPs) and Event Design Plans (EDPs), and 
gather information on tests conducted by the Anny Test and Evaluation 
Command (ATEC) using their ATEC Decision Support System (ADSS). 

• Perform electronic searches to identify reports, theses, congressional 
testimony, and periodical articles related to T&E and the CBDP. 

• Review the collected infonnation to determine whether or not each data 
type has the potential to contribute to research question answers. 

• Determine the core set of documents required for the analysis. 

• Request copies of core documents for non-CBDP programs to support a 
comparative analysis of T&E LoE for CBDP and non-CBDP programs. 

• Extract all potentially useful data elements from several documents and 
perform a preliminary analysis to determine what, if any, top-level or 
derived relationships exist within the data that can contribute to research 
question answers. 

• Finalize the data analysis approach and perform the analysis based on the 
relationships identified above. 

• Formulate conclusions and recommendations based upon the results of the 
analysis, taking the study’s limitations into account. Provide suggestions 
for future related study. 

D. ORGANIZATION OF STUDY 

This Joint Applied Project (JAP) contains the information necessary to understand 
this research effort; starting with this introduction, followed by a background section, a 
data summary, data analysis, and conclusions and recommendations. Additional 
information is included at the end of this project paper to define the acronyms used, 
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provide facts on the programs investigated, list all references used, and to define the 
initial distribution for the document. Additional details for each major chapter in this 
document are listed below. 

Chapter I: Introduction . This chapter describes the rationale behind performing 
this research study, defines the research questions guiding the effort, defines the scope 
and methodology used, defines the JAP organizational structure, and lists several 
beneficial outputs of this study. 

Chapter II: Background . Chapter II provides the basic knowledge needed to 
understand the data, analysis, and conclusions presented in this JAP. It begins with an 
introduction to the acquisition process, including definitions of the acquisition categories 
used by DoD to determine management processes. Since all but three of the programs 
studied fall within the CBDP, the history and organizational structure of the CBDP 
follows. The chapter concludes with an overview of T&E, presented in four parts; the 
basis for T&E and the TEMP, test types, program oversight, and a review of the T&E 
organizations used by the programs in this study. 

Chapter III: Data Summary . The data collected and analyzed in this JAP is 
presented in this chapter, along with a discussion that characterizes all of the data 
collected during this research The data is partitioned into three categories: principle 
data, which is the primary infonnation source for analysis in Chapter IV; secondary data, 
which only indirectly supports answers to the research questions; and unavailable, 
inconsistent, and/or deficient data, which encompasses information that either was not 
available or could not effectively be analyzed. 

Chapter IV: Data Analysis . This chapter analyzes the data presented in Chapter 
III, and presents the results obtained. The limitations and scope of the analysis are 
discussed to define the limits of the results, develops a three-pronged approach to 
execution, and then uses this approach to analyze the data. The three focus areas for the 
analysis are: system requirements, documentation, and test management scope. 
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Chapter V: Conclusions and Recommendations . The results of the analysis 
presented in Chapter IV are rolled-up into overall conclusions and recommendations in 
this chapter. The answers to the research questions defined in Chapter II are presented 
with supporting rationale. Recommendations for future academic study and for T&E 
program implementation within DoD are also presented. Finally, an overall summary of 
the research effort is presented. 

E. BENEFITS OF THE STUDY 

A PM’s ability to balance all aspects of an acquisition program to meet cost, 
schedule, and performance requirements depends upon many factors. T&E is both an 
essential risk reduction element and a large programmatic burden. Balancing program 
T&E efforts, in accordance with program size, system complexity, budget, and schedule 
is critical to the successful acquisition of new capabilities for the war-fighter. 

DoD provides T&E guidance, targeted primarily for ACAT I programs. This 
study investigates whether program requirements, documentation, and test scope are 
tailored due to a program’s ACAT, OSD/DOT&E oversight status, and materiel 
commodity area. The key outputs of this study are: a methodology that can be applied 
DoD-wide to determine technical versus operational requirements drivers for T&E, the 
characterization of the TEMP documentation burden by ACAT, insights into the effects 
of oversight on ACAT III programs, and - with the help of future research efforts - the 
ability to determine whether the T&E burden for CBDP efforts is unique or typical within 
DoD. 


5 
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II. BACKGROUND 


Galileo Galilei said, “All truths are easy to understand once they are discovered; 
the point is to discover them.” [1] While the discovery of ‘truths’ is the goal of any 
research study, the task is nearly impossible without first gaining a basic understanding of 
the area of interest. Therefore, this chapter provides the background information needed 
to understand the area of this research effort, which is Test and Evaluation (T&E) within 
the framework of the Department of Defense (DoD) acquisition process. 

This chapter begins with a top-level introduction to the DoD acquisition process 
and program acquisition category designators, followed by an overview of the Chemical 
and Biological Defense Program (CBDP) - which is the primary domain of the programs 
studied. Finally, the chapter discusses the rationale for T&E, the types of DoD T&E 
efforts, introduces program oversight, and provides a summary of the T&E organizations 
reviewed. 

A. DOD ACQUISITION OVERVIEW 

1. The Acquisition Life-Cycle Process and T&E 

The acquisition of new products within DoD has evolved significantly over the 
past few decades. Major changes include the addition of the requirement for independent 
operational test agencies within each Service; the creation of the Congressional oversight 
organization, the Director of Operational Test and Evaluation (DOT&E); and the 
sweeping acquisition reform initiatives implemented in 1994 and 1995 by the Secretary 
of Defense, Dr. William Perry. Many of these changes were targeted at improving the 
T&E aspects of the acquisition process. 

An overview of the current DoD acquisition life cycle is depicted in Figure 1. 
The acquisition of new products for DoD has proven to be a lengthy and inherently risky 
management challenge. This multi-phased life cycle approach is designed to focus 
acquisition efforts to reduce risk, with T&E a key risk reduction tool. The life cycle 
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starts with Concept Refinement (CR) and Technology Development (TD) before formal 
system acquisition begins, proceeds to formal acquisition with System Development and 
Demonstration (SDD) and Low Rate Initial Production (LRIP) followed by Full Rate 
Production (FRP), and completes with the Operations and Support (O&S) phase. A more 
detailed discussion of each phase and the top-level T&E elements in each follows. A 
brief discussion of each test type will be presented in Section A.2 of this chapter. 
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Figure 1. DoD Acquisition Life Cycle. (After: [2, p. 2-2]) 


During the CR phase, a conceptual technology approach is developed, based on 
an analysis of available technology alternatives. The Joint Requirements Oversight 
Council (JROC)-validated Initial Capabilities Document (ICD) is the requirements 
document that guides these efforts. Analysis of existing data in order to develop a viable 
approach is predominant in this early phase. Some Modeling and Simulation (M&S) and 
limited laboratory testing to prove the viability of the concept selected may also occur. 
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Milestone A (MS A) marks the end of the CR phase, and is the decision point to 
determine whether to proceed into TD. The first overarching T&E document for the 
program, the Test/Evaluation Strategy (TES), is also completed by the end of CR. 

In the TD phase, the viability of the technology approach selected is proven. 
Prototype subsystems and components are developed and technically tested, and more 
targeted M&S and laboratory testing may also be conducted to determine the limits and 
maturity of the chosen technologies. By the end of the TD phase, several key documents 
are formalized, including the first version of the program TEMP, the JROC-validated 
Capability Development Document (CDD), and an Early Operational Assessment (EOA) 
which assesses the operational impacts of the technology approach. Milestone B (MS B) 
ends this phase and is the decision point for the initiation of a new acquisition program. 
In addition, MS B marks the closure of the ability of programs to request either a partial 
or a full waiver from Live Fire T&E (LFT&E). 

After a successful MS B decision, the acquisition program officially enters the 
SDD phase. A Program Manager (PM) is assigned and delegated, through a formal 
charter, cost, schedule, and perfonnance responsibility for the program. SDD is broken 
into two sub-phases, system integration and system demonstration, with the Design 
Readiness Review (DRR) as the dividing gate. Prior to the DRR, the prototype and 
subsystem development efforts that began in TD, continue, with the goal to develop fully 
functional elements for incorporation into working system Engineering Development 
Models (EDMs). After the DRR, the EDMs are tested to determine the design’s ability to 
meet the CDD requirements. In addition to technical tests, one or more Operational 
Assessments (OA) are performed during SDD to determine whether the system has the 
potential to be both operationally effective and suitable. At the end of SDD, the TEMP 
is updated and the JROC-validated Capability Production Document (CPD) is completed. 
Milestone C (MS C) completes this acquisition phase. 

A program enters Production and Deployment (PD) and begins LRIP after MS C. 
The initial LRIP systems are primarily used for T&E, which includes IOT&E for single- 
Service programs or MOT&E for joint programs, system-level LFT&E, and technical 

tests to verify production fixes for any technical issues found with the engineering 
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development models from SDD. Later, LRIP systems are also typically fielded after 
successful completion of testing in order to provide the system’s Initial Operational 
Capability (IOC). The TEMP is updated again for the Full Rate Production Decision 
Review (FRPDR), to include any additional testing required on FRP systems, such as 
Follow-On T&E (FOT&E), as a result of tests on the LRIP systems. 

The dividing line between FRP and the final phase of acquisition, Operations and 
Support is not as clear-cut as with the previously discussed boundaries. This is due to the 
overlap of system support for early-fielded units and the production and deployment of 
systems to units with later fielding dates. FOT&E is conducted using FRP systems, if 
required, during this phase. Once all planned systems have been fielded, the Full 
Operational Capability (FOC) has been achieved. O&S, or sustainment, activities also 
include any tests required due to system problems identified after fielding, user support, 
and system disposal activities. 

2. DoD Acquisition Categories (ACAT) 

The Department of Defense organizes acquisition programs into three Acquisition 
Categories (ACATs): AC AT I, AC AT II, AC AT III. The AC AT assigned to a program is 
primarily based on the level of funding required to execute the development and 
production program activities, however program complexity and risk are other factors 
that can also drive the final ACAT determination. The ACAT assigned to a program also 
determines the types and level of reviews and the assigned Milestone Decision Authority 
(MDA), with some requirements detennined by statute. DoD also executes non-standard 
development programs, such as Advanced Concept Technology Demonstrations 
(ACTDs), Advanced Technology Demonstrations (ATDs), and Joint Warfighting 
Experiments (JWEs). A summary of the characteristics of each ACAT, including 
funding criteria, milestone review forum, and MDA is shown in Figure 2. 
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Table 3-1 

Categories of acquisition programs and milestone decision authorities 

Program category 

Program management 

Primary criteria 

($ = FY2000 constant) 

Milestone review forum 

Milestone Decision 
Authority 

ACAT 1 

ACAT ID 

PEO/PM 

more than S365M RDTE 
more than $2.19B Proc 

DAB 

DAE 

(USD(AT&L)) 

ACAT IC 

PEO/PM 

more than S365M RDTE 
more than $2.19B Proc 

ASARC 

AAE 

ACAT IA 

ACAT IAM 

PEO/PM 

excess of $32M single year 
excess of $126M total program 
excess of $378M total life-cycle costs 

DOD CIO review 

DOD CIO 
(ASD(NII)) 

ACAT IAC 

PEO/PM 

excess of $32M single year 
excess of $126M total program 
excess of $378M total life-cycle costs 

ASARC 

AAE 

ACAT II 

ACAT II 

PEO/PM 

more than $140M RDTE 
more than $660M Proc 

ASARC 

AAE 1 

ACAT III 

ACAT III 

PEO/PM 

Non-major (including C4/IT) systems 
(No fiscal criteria) 

IPR 

Designated by the 
office of the AAE 

Notes: 





1 The AAE may redelegate MDA authority at his discretion to a level no lower than the PEO (GO/SES) level. Where applicable, in the areas of medical re¬ 
search and development, Army chaplaincy or Public Affairs, the MDA may be the commander of a systems or materiel command for ACAT II and III pro- 

grams. 






Figure 2. ACAT Definitions. (From: [3]) 


An ACAT I is normally considered a Major Defense Acquisition Program 
(MDAP), and encompasses the largest defense programs. ACAT I programs are further 
divided according to specific oversight requirements and the type of program. ACAT IA 
and ACAT IAM designations are for Major Automated Information System (MAIS) 
programs. ACAT ID and IC programs differ primarily in the milestone review forum and 
the MDA assigned to the program. 

ACAT) II programs are acquisition programs that do not meet the criteria for an 
ACAT I program, but do meet the criteria for a major system, or have been designated as 
a ‘special interest’ program by the MDA. 

ACAT III programs are non-major programs that have the MDA designated by 
the AAE or CIO. These programs use In-Process Reviews (IPRs) as the milestone 
review forum. 

The majority of programs within the CBDP are ACAT Ill-level, with the larger 
programs designated as ACAT II. Other ACAT I-D programs, such as the Stryker 
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Nuclear, Biological, Chemical, Reconnaissance Vehicle (NBCRV), incorporate CBDP 
mission packages, but are still managed by non-CBDP organizations with JPEO/JPM 
support. 

B. THE CHEMICAL AND BIOLOGICAL DELENSE PROGRAM 

The overwhelming majority of programs evaluated for this project, 20 of 23, are 
managed as part of the Chemical and Biological Defense Program within DoD. 
Additionally, one of the other three programs includes a fully-integrated CBDP-managed 
mission equipment package. Therefore, a basic knowledge of the CBDP is essential to 
understanding all aspects of this report. This section outlines the history of the CBDP, 
with a focus on organizational structure, defines the commodity areas within the CBDP, 
and describes the unique aspects of the CBDP T&E management structure. 

The CBDP was established in 1994 to provide for centralized management and 
control of all DoD Chemical and Biological (CB) defense programs. This action was in 
response to Public Law No. 103-160, Section 1701 (50 USC 1522) which “mandates the 
coordination and integration of all Department of Defense CB defense programs.” [4] 
Deficiencies in DoD’s CB defense capabilities identified during Operation Desert Storm 
(ODS) provided much of the impetus for Congress to mandate improvements in CB 
defense management. 

The CBDP was initiated using the management structure shown in Figure 2. This 
organizational scheme provided DoD-level oversight of service-specific CB program 
offices. Within this structure, the Joint Service Integration Group (JSIG) was tasked with 
oversight and guidance on training, doctrine, and requirements to ensure that service- 
specific and warfighting Combatant Commanders (COCOMs) requirements and priorities 
for acquisition were met. The Joint Service Materiel Group (JSMG) was responsible for 
the CB technology, sustainment, and research and development of CB items to mitigate 
program-wide risk and to minimize duplication of efforts across commodity areas. The 
JSIG and JSMG were under to the Joint Nuclear Biological Chemical (NBC) Defense 
Board (JNBCDB) which reported to the NBCD Steering Committee. Additionally, the 
Joint Program Office for Biological Defense (JPO-BD) was linked to both the JNBCDB 
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and the medical program within DoD. Figure 3, extracted from page 23 of the General 
Accounting Office (GAO) report GAO/NSIAD-99-159, shows a slightly different view 
of the CBDP, with the inclusion of the Soldier and Biological Chemical Command 
(SBCCOM) as a complimentary material development organization to the JPO- BD. 

The CBDP commodity areas, with the lead Services designated in 1994, are listed 
below. Descriptions of all but the last commodity area are extracted from Chapter 2 of 
the 1994 CBDP Annual Report to Congress. The description of modeling and simulation 
is a CBDP-focused version of the M&S definition in DoD5000.59-M. 

• Contamination Avoidance - Army. NBC reconnaissance, detection and 
warning. 

• Individual Protection - Marines. Protective masks and protective 
clothing. 

• Collective Protection - Navy. Shelters for command posts, rest and 
relief, vehicular collective protection, and safe zones aboard ship. 

• Decontamination - Air Force. Systems used to reduce or eliminate 
hazards to units after exposure to NBC contamination. 

• Medical Defense - Army. Medical prophylaxis, pretreatment, and 
therapies used to protect personnel from the toxic or lethal effects of NBC 
threat agents. 

• Modeling and Simulation - Navy. The use of models, including 
emulators, prototypes, simulators, and stimulators, to develop data for 
making decisions related to CB defense activities. 
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Figure 3. Initial CBDP Management Structure. (From: [5]) 


Figure 111.1: Primary Planning and Executing Organizations and Programs of DOD’s Chemical and Biological Defense Program 
Research, Development, Test, and Evaluation 
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Figure 4. CBDP Planning and Executing Organizations (From: [6, p. 23]) 
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The Secretary of the Army was designated as the CBDP Executive Agent, and as 
such is “responsible to coordinate, integrate, and review all Services’ CB defense 
requirements and programs.” [7] In this organizational structure the individual Services 
executed programs to meet Service-specific and DoD requirements. In 1997 the Defense 
Threat Reduction Agency (DTRA) was established, and given responsibility for CBDP 
budgeting. 

These changes to CB defense execution within DoD were an improvement over 
previous practices but still did not vastly improve efficiencies of operation within the 
CBDP. In Figure 4, the 1999 GAO report GAO/NSIAD 99-159 recommended that an 
outcome-oriented perfonnance plan for the CBDP be developed based on the principles 
in the Government Perfonnance and Results Act (GPRA). The GAO stated that the 
“Goals of the CB Defense Program Are Vague and Unmeasurable and Do Not Articulate 
Specific Desired Impacts,” and that “CB Defense Program Performance Measures 
Emphasize Activities Rather Than Outcomes and Impacts.” Implementation of results- 
based management was inconsistent, with “The Soldier and Biological Chemical 
Command (SBCCOM)” as “the Only RDT&E Organization to Systematically Apply 
Results Act Principles.” 

These lingering deficiencies prompted major organizational changes to the CBDP 
in 2001 with the creation of the Program Executive Office (PEO) for Chemical and 
Biological Defense, which became the Joint PEO for Chemical and Biological Defense 
(JPEO-CBD) in 2003. The JPO Biological Defense functions were incorporated into the 
PEOs. This new, and also current, organizational structure for the CBDP is shown in 
Figure 5. This layout is designed to resolve issues inherent in previous CBDP 
implementations, such as single service control of programs, rampant duplicative efforts, 
disconnects between the development and test organizations, and a lack of oversight 
effectiveness. 
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This change maintains the commodity-based structure implemented in the mid-1990s, 
with two modifications; it breaks large biological detection programs into a separate JPM 
and adds a JPM focused on installation protection within the United States. Since CBDP 
funds are centrally budgeted under the defense-wide, rather than service-specific, 
category, the JPEO and JPMs manage programs that are truly joint, rather than single- 
service-driven with joint ‘potential’ or joint ‘interest.’ 

The JPEO-CBD includes eight Joint Project Managers (JPMs), primarily aligned 
by commodity areas. The JPMs with their functional areas of responsibility are: 

• JPM NBC Contamination Avoidance (Army)- NBC detection, 
obscuration, and reconnaissance 

• JPM Biological Defense (Air Force) - Biological detection 

• JPM Collective Protection (Navy) - CBR collective protection 

• JPM Individual Protection (Marines) - CBR individual protection 

• JPM Decontamination (Marines) - Individual and equipment 

decontamination 
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• JPM CB Medical Systems (Army) - Medical treatment, vaccines, and 
identification 

• JPM Information Systems (Army) - CB C4I and models and simulations 

• JPM Guardian (Army) - Installation protection 

This CBDP organizational layout centralizes management of all Service CB 
defense acquisition programs under the JPEO-CBD. This structure aligns with the 
existing designation of the Army as the Executive Agent for the CBDP, and more 
importantly, provides a single reporting chain for CBD acquisition programs within DoD. 
Requirements are handled jointly by the Joint Requirements Office (JRO) for Chemical, 
Biological, Radiological, Nuclear (CBRN) defense while CB Science and Technology is 
separately managed by the Joint Science and Technology Office (CBD) within the 
Defense Threat Reduction Agency (DTRA). Chapter 1 of the 2007 CBDP Annual Report 
provides a more detailed description of the roles and responsibilities of all CBDP 
organizations. 

Oversight of the CBDP as an entity, with the CBDP designated as an ACAT I-D 
program (non-MDAP) is provided by the Special Assistant for CBD and Chemical 
Demilitarization Programs within the Assistant to the Secretary of Defense (Nuclear, 
Chemical, and Biological Defense Programs) office. [9] Additionally, individual 
programs may also be under DOT&E oversight and/or be designated as Defense 
Acquisition Executive (DAE) ‘Sentinel’ oversight programs within the CBDP. 

For T&E, the Deputy Under Secretary of the Army (Operations Research) 
(DUSA(OR)) has responsibility as both the Army and the CBDP T&E executive. This 
enables the Secretary of the Army to hold both the materiel development executive, the 
JPEO-CBD, and the test executive, the CBDP T&E Executive, accountable for programs 
within the CBDP. 

Another CBDP-unique organization is the Product Director of Test Equipment, 
Strategy, and Support (PD-TESS) within the office of the JPM NBC Contamination 
Avoidance. The PD TESS is chartered “to support the CBDP T&E Executive in matters 
of test infrastructure development.” [8, p. 63] The PD TESS is tasked to manage 
upgrades to the CBDP test infrastructure (e.g., CB agent test facilities) through FY11 to 

address existing test facility limitations to effective CB test execution. 
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In summary, the one and a half billion dollar per annum CBDP is a 
congressionally-mandated, uniquely-managed, joint enterprise within the DoD. It has 
taken a decade for DoD to implement an effective management and oversight structure 
for the CBDP, to include a separate CBDP T&E Executive, an organization dedicated to 
test infrastructure improvement, and to provide acquisition program responsibility to a 
single truly joint organization. 

C. INTRODUCTION TO DOD TEST AND EVALUATION 

1. The Basis for Test and Evaluation and the TEMP 

The two basic reasons that T&E is perfonned are: to meet legal and procedural 
requirements, and to determine whether a system can meet the performance requirements 
levied upon it. While acquisition programs develop many test-related documents, it is the 
TEMP that is the keystone for program T&E efforts. 

a. Legal and Procedural Basis for T&E 

The formal requirement to perform T&E within DoD is derived from 
multiple sources, including public laws, statutes, executive orders, and public policies. 
DoD uses these sources to develop the policy memorandums, DoD Directives (DODDs), 
Instructions (DODIs), and Regulations (DOD-Rs) that implement these requirements. 
Per DoD Directive 5000.1, it and DoD Instruction 5000.2, define “management 
principles and mandatory policies and procedures for managing all acquisition 
programs.” These details include definition of key T&E activities and documents, and 
form the basis for the development of lower-level documents, such as service-specific 
policies and implementation guidance. Examples of Service-specific T&E 
implementation documents include Army Regulation 73-1, DA Pamphlet 73-1, Air Force 
Policy Directive 99-1, Secretary of the Navy Instruction 5000.2, and Marine Corps order 
3960.2B. 
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b. Operational Basis for T&E 


Even if the legal and procedural requirements did not exist, there still 
would be a driving force behind the need for T&E - verifying that a system meets the 
user’s operational requirements. A system must provide a technical capability that the 
end user can utilize to meet a mission need, i.e., the system must be operationally 
effective and suitable for the warfighter. The user-driven need for T&E is clearly 
summarized by 

T&E must demonstrate capabilities today that will be needed tomorrow 
in combat. When a system is called for combat duty, the need is 
immediate and there’s no time to reconsider if the system will operate as 
designed. Warfighters need assurance that when they risk their lives, the 
systems provided will do the job for which they were designed. [10, p. 7] 

c. The Role of the TEMP in T&E 

DoD Instruction 5000.2 defines the requirement for a TEMP, and per 
Table E3.T2., requires completion of the initial version of a TEMP for an acquisition 
program by MS B, with updates for the MS C and the FRP decisions. DA Pamphlet 73-1 
summarizes the role of the TEMP as “the basic planning document for all life cycle T&E 
related to a particular system acquisition and is used by decision-making bodies in 
planning, reviewing, and approving T&E activities.” [11, p. 3] The DA Pamphlet further 
defines the TEMP as the “overarching T&E document used by the T&E community to 
generate detailed T&E plans and to ascertain schedule and resource requirements 
associated with the T&E program.” [11, p. 3] 

The Program Manager (PM) is tasked with development of the TEMP, and 
specifically with the technical, or developmental, T&E objectives included in the TEMP. 
The Operational Test Agency (OTA) is responsible for developing the OT&E objectives 
within the TEMP. Each system objective is described in the TEMP, typically with 
technical parameters called Critical Technical Parameters (CTPs) and operational 
parameters called Critical Operational Issues (COIs), Measures of Effectiveness (MOE), 
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Measures of Suitability (MOS), or Measures of Performance (MOP). Objectives 
categories may also include items such as system survivability and interoperability. 

Per DA Pamphlet 73-1, CTPs “are measurable critical system 
characteristics that, when achieved, allow the attainment of operational performance 
requirements.” [11, p. 4] The key word in this phrase is ‘allow,’ which shows that CTPs 
define the technical capability that is required to make the desired operational result 
possible. 

In contrast, operational test parameters are directly based upon the COIs, 
COI and Criteria (COICs), and Key Performance Parameters (KPPs) from the system’s 
requirements document, from which the operational test evaluation requirements, the 
MOE, MOS, and MOP, are derived. 

Two types of performance measures, technical and operational, are the key 
parameters used during T&E efforts to determine whether a system can meet the user’s 
operational requirements. Although each TEMP is program-specific, all “shall describe 
planned developmental, operational, and live fire testing, including measures to evaluate 
the performance of the system during these test periods; and integrated test schedule; and 
the resource requirements to accomplish the planned testing.” [12, p. E5.1.4.1] Since the 
TEMP must include all of the items necessary to execute the T&E efforts, it, therefore, is 
the one acquisition document that provides the infonnation necessary for a comparative 
analysis of programs across acquisition categories, functional areas, and time periods. 

2. Types of DOD and Commercial Testing 

In this age of rapidly-advancing technologies, military weapon systems, air and 
ground vehicles, and support equipment are becoming increasingly difficult to acquire, 
maintain, and deploy into a combat environment. The demand for SMART technology, 
super battle-hardened equipment, and embedded software, tangled with millions of line 
of code, pushes the envelope of defense system complexity. The resulting increased risk 
to cost, schedule, and performance baselines will have a momentous effect upon the way 
DoD conducts T&E to verify technical parameters and validate operational perfonnance. 
Currently, DoD conducts single-tiered, event-driven Developmental Testing (DT) and 
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Operational Testing (OT) overlapping with the contractor testing activities. Embedded 
within DT and OT is a host of pre and post-Full Rate Production (FRP) sub-testing 
events. Additionally, many program offices are choosing combined DT/OT, streamlining 
the T&E activities in efforts to mitigate cost and schedule risk. Regardless of the 
strategies employed, program offices must adhere to applicable laws, regulations, and 
oversight authorities when planning and conducting the numerous T&E efforts in support 
of an acquisition program. Consequently, this rigorous and expensive process enables the 
fielding of reliable products and promotes user confidence. 

Modern developmental testing assesses the accomplishment of meeting critical 
technical parameters and determines system readiness to proceed to the Initial 
Operational Testing (IOT). [13, p. 16] Shown in Figure 1, DT is typically conducted 
throughout the Conceptual Refinement, Technology Development, and System 
Development and Demonstration phases, even continuing up through FRP, when 
involved in an Evolutionary Acquisition (EA) process. DT is an engineering tool that 
supports critical decisions on the choices of design plans, materials, manufacturing 
processes, and new or legacy technology integration. Contractor testing is typically 
conducted during the Conceptual Refinement and Technology Development phases, and 
continually monitored by the program office in preparation for testing and entrance 
criteria for major milestone decisions. Essentially, developmental testing minimizes 
design and integration surprises, which in some cases, can be fatal to an acquisition 
program. These pre-FRP tests are termed Product Verification Test (PVT), Production 
Qualification Test (PQT), Limited User Test (LUT), Live Fire Test (LFT), and the special 
testing requirements for the JPEO-CBD programs, Live Agent Test (LAT). For a 
description of these and other types of pre-FRP testing listed below, refer to Army 
Regulation 73-1. 

• Research effort or test 

• Technical feasibility test 

• Engineering development test 

• Production prove-out test (PPT) 

• Software qualification test 
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• Live fire test for vulnerability & lethality 

• Logistics demonstration 

• C4I/IT interoperability certification test 

• Software development test (SDT) 

Operational testing provides qualitative and quantitative data for system 
assessments and evaluations, and determines operational effectiveness, suitability, and 
survivability of the system under realistic battlefield conditions. OT is controlled by an 
independent agency, using production-representative systems, while allowing restricted 
contractor involvement to minimize bias, and essentially supports a decision to pass or 
fail a defense system before it goes into full-scale procurement. 

Refer to AR Regulation 73-1 for a description of the various operational tests. 

• First article test 

• Comparison test 

• Live Fire Test, if required for product improvements of covered systems 

• Quality conformance (acceptance) inspection 

• Tests in support of post-deployment software support (PDSS) 

• C4I/IT interoperability recertification test 

• Surveillance test. 

• Reconditioning test. 

A unique set of requirements for Chemical, Biological, Radiological, and Nuclear 
defense systems is Live Agent Testing. LAT refers to the introduction of weapons-grade 
chemical or biological contaminants, also known as Weapons of Mass Destruction 
(WMD) in solid, liquid, and vaporous forms to the system being tested. Detection 
systems’ performance is quantitatively characterized by measurements associated with 
the speed and rate of detection and agent concentration. Additionally, the introduction of 
common battlefield materials, such as fuels, solvents, lubricants, smokes and obscurants, 
etc., detennines the detector’s potential to false alann. 

Testing collective protection systems, such as filtration equipment, in mobile 
laboratories, shelters, and combat vehicles determines the system’s ability to safely 
collect chemical and biological agents while providing breathable air to the occupants. 

Testing individual protection systems, such as Personal Protective Equipment (PPE), i.e. 
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masks, gloves, boots, and protective clothing determines if the product adequately shields 
the wearer from the agent for a given mission time requirement. Operational testing with 
live agents is neither practical, safe, nor legal in the continental United States. Therefore, 
testers and evaluators use simulants for field testing with actual users. Simulants are an 
effective testing method for safely replicating CBRN environments in military 
operations. 


3. Program Oversight 

One of the factors affecting many programs is oversight. Oversight is “Senior 
Executive-level monitoring and review of programs to ensure compliance with policy and 
attainment of broad program goals.” [10, p. 66] Programs can be designated for 
oversight based on statutes and regulations, and by designation by organizations such as, 
the Under Secretary of Defense (Acquisition, Technology, and Logistics) (USD(AT&L)), 
DOT&E, or a Service. Oversight can be specific to one aspect of a program, such as 
T&E, or general in nature, where all key program documents and decisions require 
oversight-organization approval. The two types of oversight evaluated during this study 
were, the Office of the Secretary of Defense (OSD) T&E oversight/DOT&E oversight 
and CBDP-specific oversight. 

OSD oversight is primarily focused on T&E, and therefore is tightly coupled with 
the congressionally-mandated DOT&E oversight function. DOT&E ultimately reports to 
Congress, but is organizationally situated within DoD. Within USD(AT&L), the 
Director of DT&E/Office of Defense Systems (DS) designates programs for OSD T&E 
oversight jointly with DOT&E, in consultation with Service (and CBDP) T&E 
executives, per DODI 5000.2. The 2007 OSD T&E oversight list, although marked as 
For Official Use Only (FOUO), is publicly accessible at the following website, http, 
//www.acq.osd.mil/sse/as/docs/2007_TE_Oversight_List.pdf. The OSD oversight list 
includes ACAT I and IAM, II, III, special interest, pre-MAIS, and pre-MDAP programs, 
and therefore OSD/DOT&E oversight is not unique to large programs. 

Another type of oversight, called “Sentinel,” existed within the CBDP from April 
2003 through September 2007. Sentinel was a program-wide oversight activity 

performed by the USD(AT&L) on CBDP advanced development programs that had the 
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cost, complexity, and criticality sufficient to warrant monitoring the program’s cost, 
schedule, and perfonnance “as an indicator of the general programmatic health of the 
functional area.” [9] Sentinel oversight and OSD oversight were not mutually exclusive. 
Therefore, some programs within the CBDP had both types of oversight simultaneously. 
Sentinel oversight is not separately evaluated in this study, as all Sentinel systems 
reviewed were already on the OSD T&E oversight list. 

4. Testing and Evaluation Organizations 

Today, many DoD acquisition programs utilize numerous testing options 
available from the different services and private industry. While a substantial portion of 
developmental testing is the responsibility of the contractor, the majority of 
developmental and operational testing and evaluation of DoD systems occurs at military 
T&E facilities. The Department of Defense acquisition program offices are responsible 
for early coordination with these organizations, and typically dedicate a portion of the 
office staff to the responsibility of managing the program test and evaluation activities. 
The areas of responsibility characteristically includes defining test scope and objectives, 
documentation, and T&E funding, in addition to coordinating all of the test resources 
throughout each phase of the acquisition process. [2, p. 4-1] Consequently, the PMO’s 
dedicated T&E activity must possess a comprehensive knowledge of the system and T&E 
process, and discuss alternatives associated with selecting the appropriate support activity 
for an effective test and evaluation strategy. 

This section familiarizes the reader with various test and evaluation activities 
throughout DoD, however, limited to those agencies covered under the Data Summary 
and Data Analysis chapters. 

a. Army T&E Agencies 

The Army Test and Evaluation Command (ATEC) is headquartered in 

Alexandria, Virginia, and is DoD’s single-most integrated testing and evaluation 

operation. ATEC comprises integrated evaluations conducted at the U.S. Army 

Evaluation Center (AEC), developmental testing at the U.S. Army Developmental Test 

Command (DTC), and operational testing at the U.S. Army Operational Test Command 
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(OTC). On October 1, 1999, the Army finalized the earlier decision to move 
developmental and operational evaluation into a single, integrated command. Figure 6 
represents the Anny’s current organizational chart that identifies the major commands 
and subordinate centers. 

The Test and Evaluation Command (TECOM) became a major 
subordinate command of ATEC and was redesignated the U.S. Army 
Developmental Test Command (DTC), with DTC headquarters remaining 
at Aberdeen Proving Ground, Maryland. Also, the Test and 
Experimentation Command (TEXCOM) was redesignated the U.S. Army 
Operational Test Command (OTC), with OTC headquarters remaining at 
Fort Hood, Texas. The third ATEC subordinate command that was 
redesignated encompassed both the Operational Evaluation Command and 
the Evaluation Analysis Center, which were combined to form the new 
U.S. Army Evaluation Center (AEC). [14] 

DTC is headquartered at the Aberdeen Proving Ground (APG), Maryland, 
and is responsible for the planning and management activities of all developmental 
testing conducted at its subordinate centers. DTC also offers testing services to other 
DoD military services and federal agencies, state and local governments, foreign and 
allied governments, and private industry. OTC is headquartered at Fort Hood, Texas, and 
is responsible for the planning and management of all operational testing in the areas of 
equipment, doctrine, force design, and training, using typical soldiers to determine 
system effectiveness, suitability, and survivability. 

The Army Evaluation Center (AEC) is headquartered in Alexandria, 
Virginia, while maintaining locations at APG, Fort Monmouth, New Jersey, and Fort 
Bliss, Texas. AEC is the Army’s total system evaluator who provides to the final 
decision-makers the documentation on system effectiveness, suitability, and overall 
performance during testing. AEC’s early involvement in the acquisition process and 
working closely with the PMs and test communities ensures that T&E strategies and 
objectives are consistent throughout the acquisition program. 

The Army Materiel Systems Analysis Activity (AMSAA) is part of the 
Research, Development, and Engineering Command (RDECOM) and is located at the 
Aberdeen Proving Ground. AMSAA provides the DoD acquisition and T&E 
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communities with analysis support and solutions to materiel and logistics systems issues 
to support the decision-making process for acquiring and fielding new technologies. 

The Army Research Laboratory (ARL) is organized under the Research, 
Development, and Engineering Command (RDECOM), and is headquartered at the 
Adelphi Laboratory Center in Adelphi, Maryland. ARL provides the Army and other 
DoD organizations with applied research, analysis, and testing of weapons, materials, 
human systems integration, computational sciences, and vehicle and equipment 
survivability and lethality. 

The Aberdeen Test Center (ATC) is located at the Aberdeen Proving 
Ground, 25 miles northeast of Baltimore, Maryland, bordering the Chesapeake Bay. The 
installation consists of 79,000 acres of water and land mass, and has been operating since 
January 1918. Today, ATC is the lead DoD test center for automotive (manned and 
unmanned ground vehicles), large caliber and small anns, direct fire and munitions, and 
live fire vulnerability and lethality testing. 

The Aviation Technical Test Center (ATTC) is located at Cairns Army 
Airfield, Fort Rucker, Alabama. ATTC is the U.S. Army’s flight test activity for 
airworthiness-qualification of both rotary and fixed-wing aircraft, aviation systems, and 
aviation support equipment. 
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Figure 6. U.S. Army Test and Evaluation Command Organization Chart (From: [14]) 


Dugway Proving Ground (DPG) is home to the West Desert Test Center 
(WDTC), and is located 85 miles southwest of Salt Lake City, Utah, in the Great Salt 
Lake Desert on 798,855 acres of land. WDTC serves as the nation’s chemical and 
biological defense proving ground, and also conducts reliability and survivability testing 
for meteorological, smoke, obscurants, illumination, and munitions systems. 

The Yuma Proving Ground (YPG) and the Yuma Test Center (YTC) is an 
ATEC test activity located near the Arizona-California border, approximately 24 miles 
north of the city of Yuma, Arizona. In the context of this JAP, the reference of YPG or 
YTC is considered to be the same test activity. YTC is a multi-purpose test complex for 
the Army’s desert environment testing, with only about three inches of rainfall per year, 
encompassing more than 1,300 square miles of test range area and 2000 square miles of 
restricted airspace. The test center offers realistic testing in air delivery, ground combat 
systems, aircraft armament, unmanned systems, and automotive commodity areas. 

The Redstone Technical Test Center (RTTC) is one of the U.S. Army Test 
and Evaluation Command (ATEC) activities located on Redstone Arsenal near 
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Huntsville, Alabama, and encompasses over 14,000 acres of land. RTTC is one of the 
eight test centers that comprise the Developmental Test Command. RTTC’s primary 
service is to provide advanced testing of weapon systems for the Department of Defense 
and its contractors. The Center provides advanced testing in two major areas; flight¬ 
testing of small rockets and guided missiles and advanced life-cycle testing for weapon 
subsystems and components. 

The Tropic Regions Test Center (TRTC) is located in Hawaii and Panama, 
while other test facilities are currently being established over a wide geographic area for 
growing tropical testing requirements. TRTC offers real-world tropical effects that 
include insects, bacteria, high temperature and humidity conditions, RAM testing of 
soldier systems, and provides mobility testing to evaluate system ruggedness and small 
team effectiveness. 

The Cold Regions Test Center (CRTC) is located in the mountainous 
regions of Alaska encompassing over 670,000 acres of land with restricted airspace to 
unlimited altitude. CRTC accommodates a full range of cold weather or temperate 
climate tests, including automotive cold start capabilities, mines, explosives, small arms 
tests, direct fire tests, sensor testing, air defense, missile, artillery, smoke and obscurant 
tests, and mobility testing. CRTC can accommodate indirect fire testing with the 
capability of observed fire to 30 km and unobserved fire to 50 km. Indirect fire up to 100 
km is accomplished by utilizing ranges near Fort Wainwright, with the impact on Fort 
Greely areas. 

The Anny’s Electronic Proving Ground (EPG) is located at Fort 
Huachuca, Arizona, and encompasses 76,000 acres of land. EPG’s remote location is an 
environment free of radio-frequency interference, making it the Army's nucleus for 
testing Command, Control, Communications, Computer, and Intelligence (C4I) 
equipment and electronic systems. EPG offers developmental testing of C4I systems, 
Unmanned Aerial Vehicles (UAVs), and navigation and avionics systems. 

The White Sands Missile Range (WSMR) is located 20 miles east of Las 
Cruces, New Mexico, and is where the United States successfully tested the world's first 
atomic bomb. WSMR covers 3,200 square miles, which makes it the largest military 
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installation in the country. WSMR offers a variety of services including research and 
development, military systems and commercial product assessments, environmental 
testing, computer modeling, and open-air/over-land missile testing. 

Fort Indiantown Gap National Guard Training Center (FITG) is located in 
Annville, Pennsylvania, 23 miles east of Harrisburg, Pennsylvania. FITG is a 17,000 
acre National Guard Training site, and offers military and law enforcement training, 
equipment testing, and accommodates a variety of live-fire ranges, including small arms, 
shoulder-launched rockets, and anti-armor devices. 

b. Navy and Marines T&E Agencies 

The Operational Test and Evaluation Force (COMPTEVFOR) is located at 
the Norfolk Naval Base in Norfolk, Virginia. COMPTEVFOR is the Navy’s sole 
independent test and evaluation activity, reporting directly to the Chief of Naval 
Operations. Slightly dissimilar to the Army’s T&E organizational structure, 
COMPTEVFOR combines test and evaluation within each division, and structured along 
flexible lines, which gives primary consideration to the type of warfare and expertise. 
These areas are the Undersea Warfare Division, Aviation Warfare Division, C4I and 
Space Division, Surface Warfare Division, and Expeditionary Warfare Division. Unless 
otherwise specified, the use of COMPTEVFOR in this JAP refers to the Navy evaluation 
support. Reference to other Navy activities constitutes testing support. 

The Marine Corps Operational Test & Evaluation Activity (MCOTEA) is 
located at Quantico, Virginia, and is the United States Marine Corps (USMC) OT&E 
authority for all Marine systems and equipment. 

The Naval Research Laboratory (NRL) is located in the southwest 
quadrant of Washington, DC. NRL conducts an extensive multidisciplinary program of 
technological development, scientific research, and collaborative testing associated with 
maritime applications including advanced materials, support equipment, atmospheric 
systems, and space sciences. 

The Naval Air Systems Command (NAVAIR) is headquartered at the 
Patuxent River Naval Base in Patuxent River, Maryland. NAVAIR is one of the Navy’s 

five systems commands and designs, acquires, manages, and maintains the Navy air 
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systems enterprise. The Naval Air Warfare Center Aircraft Division (NAWCAD), also 
located in Patuxent River, is one of the subordinate organizations of NAVAIR. 
NAWCAD executes life-cycle support in the areas of engineering development, 
acquisition, test and evaluation, logistics, and training. In addition, the Fleet Readiness 
Center East, at the Marine Corps Air Station Cherry Point, North Carolina, is a member 
of the NAVAIR community, primarily providing the Marine Corps and Navy with 
aviation maintenance and RDT&E support. Furthermore, Cherry Point supports DoD 
with logistics management and direct field support for a variety of assignments. 

The Naval Sea Systems Command (NAVSEA) is headquartered in 
Washington, DC, and is the largest of the Navy’s systems commands. The NAVSEA and 
NAVAIR organizations function is a similar way, where NAVSEA is responsible for the 
Navy’s entire fleet and their combat systems. The Naval Surface Warfare Center 
(NSWC), Dahlgren Division, is located in Dahlgren, Virginia, and is structured under 
NAVSEA. Dahlgren primarily focuses on research, development, and testing on warfare 
systems, weapons, munitions, sensors, software, and force protection. Dahlgren also 
offers drop testing, environmental effects, and electromagnetic energy interference 
testing and evaluation. The Carderock Division is located in West Bethesda, Maryland, 
focusing on survivability, structures, and materials. Carderock also specializes in 
equipment shock hardening, vibration testing, hull design, and shipboard survivability. 
The Navy Clothing and Textile Research Facility (NCTRF) is a NAVSEA asset for the 
RDT&E of personal protective equipment, damage control, and fire protection 
engineering. NCTRF hosts a variety of testing capabilities, including a dynamic fire test 
chamber, a pressurized steam test chamber, and a thennal oven chamber specifically 
designed for the testing of PPE. The Navy’s Space and Naval Warfare Systems 
Command (SPAWAR) is located in San Diego, California, and provides DoD with 
RDT&E and logistical support, acquisition management of joint systems associated with 
C4I and space technologies. 

The Naval Operational Medicine Institute (NOMI) is located in Pensacola, 
Florida. NOMI participates in doctrine development, experimentation, and testing in 
support of naval operations and naval survival systems. 
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The Naval Safety Center (NSC) is located in Norfolk, Virginia, and 
provides the Secretary of the Navy (SECNAV) with safety-related management for all 
Naval and Marine Corp issues. NSC also provides the DoD acquisition community with 
system evaluations, risk identification, and hazard analysis documentation for programs 
under development. 

Naval Station Mayport, Florida, is adjacent to Jacksonville and situated on 
3,400 acres of land. Mayport provides naval fleet port accommodations, operational and 
training activities for anti-submarine warfare, and also provides on-board operational 
testing and training support through the Afloat Training Group (ATG). 

The Marine Corps Systems Command (MARCORSYSCOM), 
headquartered at Marine Base Quantico, Virginia, and the home of Marine training and 
doctrine since 1917. MARCORSYSCOM is the principal activity for acquisition, testing, 
and sustainment of systems and equipment used by the Marine forces. 
MARCORSYSCOM products encompass the full spectrum of warfighter needs, 
including armor, combat and support, C4I, ground transportation, and sensor systems. 

c. Air Force T&E Agencies 

The Air Force Operational Testing and Evaluation Command (AFOTEC) 
is headquartered at Kirtland Air Force Base (AFB), New Mexico, which is also the site to 
one of five Air Force T&E detachments; Eglin AFB-Florida, Peterson AFB-Colorado, 
Edwards AFB-California, and Nellis AFB-Nevada. AFOTEC provides multi-service test 
and evaluation support in the areas of weapons, support equipment, C4ISR, space, missile 
defense, and aircraft. 

The Air Force Medical Operations Agency (AFMOA) is located at Brooks 
Air Force Base, San Antonio, Texas. AFMOA is responsible for the Air Force’s 
aerospace medicine, bioenvironmental engineering and bioenvironmental, and is the 
consultant lead for aerospace medicine and preventive medicine, clinical research, 
medical resources, and chemical, biological, and radiation protection. 

Edwards Air Force Base, California, is home to the Air Force Flight Test 
Center (AFFTC). AFFTC conducts flight and ground testing of aircraft and support 
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equipment, weapons systems, software, and additionally perfonns Modeling and 
Simulation (M&S) for the Air Force. 

Wright-Patterson Air Force Base (WPAFB), Ohio, is home to the Air 
Force Materiel Command (AFMC) conducting RDT&E, acquisition management, and 
logistical operations for Air Force systems. WPAFB is also home to the Air Force 
Research Laboratory (AFRL), Aeronautical Systems Center, and National Museum of the 
United States Air Force. 

Prior to its closure in 2001, McClellan Air Force Base in Sacramento, 
California, was home to the Sacramento Air Logistics Center, situated on 3,778 acres of 
land, and conducts a variety of missions including the maintenance and management of 
US Air Force electrical components, communications, electronics systems, fluid drive 
accessories, and tactical shelters. 

Hill Air Force Base is located in northern Utah, operating and maintaining 
the Utah Test and Training Range. The range provides open air testing services for a 
variety of needs, including weapons testing, large force training exercises, and range 
support for Air Force operational test and training programs. 

The Air Mobility Command at Pope Air Force Base, North Carolina, is 
responsible for delivering humanitarian effects and expeditionary aid, aerial refueling, 
and aerial combat support. Additionally, Pope AFB groups provide support aircraft to 
training exercises and operational testing. 

d. The Commercial Sector 

Wyle Laboratories Inc. is headquartered in El Segundo, California, with 
more than 30 facilities nationwide. Wyle labs provide test and evaluation support to 
DoD, NASA, and commercial customers with vibration systems testing, environmental, 
and structural testing. Wyle also provides support in acquisition management, 
engineering, and space sciences. 
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Alion Science & Technology R&B Laboratory operates as an independent 
testing lab for electro-magnetic emission and interference, and radio frequency 
quantification supporting the automotive industry, various commercial aircraft and 
avionics products, and military applications. 

Battelle is a large international science and technology business, 
headquartered in Columbus, Ohio, possessing RDT&E capabilities in energy 
conservation, lab management, heath sciences, and national security and defense. 
Additionally, Battelle supports defense projects associated with chemical and biological 
detection and protection equipment for the joint services. 

Lincoln Laboratory at the Massachusetts Institute of Technology (MIT) is 
a federally funded R&D center for applied science and technology, specializing in 
advanced electronics applications associated with air and missile defense, 
communications, tactical and space surveillance, and air traffic control systems. 
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III. DATA SUMMARY 


A. INTRODUCTION 

The researcher’s approach to data collection is to gather information for the 
purposes of both elimination and discovery. This approach is based upon a philosophy 
that there are no erroneous samples of data. The many significant discoveries that have 
been made by exploiting seemingly useless data for other purposes, supports this mindset. 
The tester’s approach to data collection is more narrowly focused; to validate a known or 
desired parameter through measurement. 

Data was collected for this research project using the philosophy of a researcher, 
with the constraint that each data element must be applicable to multiple programs. This 
constraint minimized the collection and presentation of data that had no possibility of 
helping to answer the primary and secondary research questions. The data presented in 
this chapter was collected from twenty-three acquisition programs. These programs 
cover the breadth and depth of acquisition possibilities, including, ACAT I, II, and III; 
T&E oversight and non-oversight; joint service and Army-only; CBDP and non-CBDP; 
Milestone B through Full Rate Production; and both current and legacy acquisition 
programs. 

The majority of the data contained in this chapter is directly traceable to data 
elements extracted from the twenty-three program TEMPs recorded in the List of 
References. Other documents, such as the program’s performance specification were 
used to fill TEMP data gaps when possible. The one derived category of data element 
presented in this chapter is the quantifiable data for CTPs and MOE-MOS. The 
derivation method for the quantifiable data types is defined in Chapter IV. 

The data presented in this chapter is categorized as, principal data; secondary 
data; and unavailable, inconsistent, and/or deficient data. Principal data is the primary 
information source for analysis in Chapter IV. Secondary data only indirectly supports 
answers to the research questions, while the last category of data encompasses 
information that either was not available or could not be effectively analyzed. 
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B. DATA PARAMETERS 


1. Principal Data 

Principal data is defined as tangible and consistently available information 
associated with each program. This infonnation provides administrative and technical 
details for PM planning and estimating purposes, and is equally valuable for strategic 
T&E preparation. 

Programmatic and Technical data consists of Critical Technical Parameters, 
Measures of Effectiveness, and Measures of Suitability. CTPs describe system hardware 
and software characteristics and thresholds contained in the CDD, and must be consistent 
with system operational performance requirements. Critical technical parameters are, 

measurable critical system characteristics that, when achieved, allow the 
attainment of desired operational performance capabilities. They are not 
user requirements. Rather, they are technical measures derived from 
desired user capabilities. Failure to achieve a critical technical parameter 
should be considered a reliable indicator that the system is behind in the 
planned development schedule or will likely not achieve an operational 
requirement. [15, p. 9.10] 

Table 1 presents the total number of CTPs and quantifiable CTPs for each program. 

MOEs are assessments of operational performance that evaluate the degree of 
successfulness of the system with respect to the objective of the mission or operation. [2, 
p. B-11] Specifically, the MOE assesses the mission accomplishment of a system when 
used by representative personnel in the environment planned or expected (e.g., natural, 
electronic, threat etc.) for operational employment of the system considering 
organization, doctrine, tactics, survivability, vulnerability, and threat. [15, p. 9.10] A 
MOS is an operational assessment that evaluates a system’s capacity of being 
satisfactorily fielded in an environment considering issues pertaining to reliability, 
availability, maintainability, logistical support, compatibility, interoperability, training, 
human factors, safety, documentation, transportability, wartime usage rates, manning 
requirements, and natural and environmental effects and impacts. [15, p. 9.10] Table 2 of 
this chapter presents MOE, MOS, and quantifiable MOE/MOS data sets. The Defense 
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Acquisition Guidebook suggests that program managers must develop a crosswalk for 
operational effectiveness and suitability parameters and constraints 

to those used in the Analysis of Alternatives (AoA), and include 
manpower, personnel, training, software, computer resources, 
transportation (lift), compatibility, interoperability and integration, 
Information Assurance (IA), Electromagnetic Environmental Effects, and 
Spectrum Supportability. The focus is on operational capabilities, not 
design specifications such as weight and size. [15, p. 9.10] 

Although seemingly trivial, the TEMP page count, Table 3, is essentially the total 
number of pages, excluding the cover and signature pages, and the number of pages in 
sections I through V. Surprisingly, this information gives useful insight into which 
programs followed the DoD recommended TEMP page count guidelines and fonnulates 
insight into the level of documentation effort. 

Many DoD acquisition programs utilize numerous organizations for T&E support 
depending upon system requirements, organizational past perfonnance, and range 
availability, to name a few. The programs investigated in this JAP benefited from the 
diverse expertise of multi-service and private testing organizations briefly introduced in 
Chapter II, consisting of the Army, Navy, and Air Force testing and evaluation 
commands, in addition to a host of industry and university locations, such as Lincoln 
Labs at MIT. Tables 5 through 12 illustrate the various test and evaluation organizations 
utilized by each program. 

2. Secondary Data 

Selected programmatic and service-specific information concerning each program 
was collected for reference and to help bridge gaps in formulating an enhanced 
comprehensive analysis. This section consists of the total number of estimated personnel 
required throughout DT and OT, COIs, the TEMP date, and program oversight 
requirements. 
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The TEMP date, shown in Table 4, is the official date of document authorization 
and/or the current documented revision. Each program’s TEMP date was collected for 
comparative purposes in an attempt to discover a relationship of pre/post-year 2000 
TEMPs to other principal and secondary data. 

All ACAT I, and selected ACAT II programs, require OSD and DOT&E 
oversight as described in the Background Chapter. However, many ACAT III programs 
are not burdened with this acute, yet necessary supervision. Three of the seventeen 
ACAT III programs investigated required some level of oversight. Interestingly, these 
three programs are detection systems. 

Critical Operational Issues are the key operational effectiveness or suitability 
issues that must be scrutinized during OT&E to determine a system’s capability of 
successfully executing the intended mission. COIs are typically represented as questions. 
The total number of COIs for each program are listed in Table 2. The number of COIs 
demonstrated no stand-alone significance. However, the development of a mathematical 
analysis of effectiveness and suitability issues enabled a direct comparison of programs 
across ACAT levels and is investigated in Chapter IV. 

3. Unavailable, Inconsistent, and/or Deficient Data 

Certain program data was initially perceived as indisputable evidence supporting 
the questions in this JAP. The most difficult and disappointing sets of data was program 
funding. The original methodology was to focus all other supporting data with the 
amount of funding each program received and detennine if the smaller programs 
typically endured the same T&E burden while having less funding and resources. It was 
discovered that the funding requirements estimated in the TEMPs only reported 
combined RDT&E funding. A valid funding comparison was not possible due to the 
drastic differences in system complexity and varying production quantities. High 
complexity programs with less technical maturity will likely have more research funding. 
This budgeting information must be a separate line item of the total RDT&E estimate in 
order to accurately detennine the T&E segment. Additionally, the RDT&E funding 
reported in the Program Objective Memorandum (POM) drastically differed from the 
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data gathered from TEMPs. For example, POM funding data includes congressional 
plus-ups and reductions, program re-structuring, and out of control cost increases, 
rendering a sensible comparison virtually unattainable. Additionally, the TEMP funding 
data were skewed based upon the various acquisition phases. 

Another data set expelled from the analysis was the total number of required 
personnel estimated throughout testing due to incompleteness and inconsistencies across 
the programs. Some TEMPs were exceedingly specific, listing the various multi-service 
requirements, logistical support personnel, the number of evaluators, etc. Some TEMPs 
explicitly called for precise numbers of officers and enlisted personnel by their actual 
Military Occupational Specialty (MOS), the number of battalions, companies, teams, 
shipboard and Operational Force (OPFOR) participants during specific events. Other 
programs were exceedingly broad in their requirements, merely mentioning the need for 
test support personnel and OT participants. 

Likewise, an impartial comparison of the number of test articles and supporting 
equipment estimated in the program TEMP’s emerged from the data set. It is not 
practical to compare a program requiring nine vehicles to a chemical detection system’s 
requirement of 250 detectors, or even 5,750 masks for a mask program. Additionally, 
some programs were very specific in listing supporting equipment necessary during 
testing. For example, the Joint Service General Purpose Mask (JSGPM) program 
required a video reduction station, one portable weather station, two thousand vision 
correction systems, various video and audio components, and twenty mask leakage 
testers. In contrast, the Joint Chemical Agent Detector (JCAD) program required 
numerous vehicles, including sixteen High Mobility Multipurpose Wheeled Vehicles 
(HMMWVs), three Bradley Fighting Vehicles (BFV), six utility vehicles, and six ten- 
passenger vans. The JCAD TEMP merely mentioned the need for 35 handheld radios 
and two Single-Channel Ground-Air Radio System (SINGARS) radio sets. This disparity 
in requirements was consistently evident across the entire set of TEMP documents and 
did not demonstrate the continuity required for the analysis in Chapter IV. 

The TEMP milestone, shown in Table 4, is the major milestone at which the 

program achieved authorization, or the current published document version. Knowing 
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the acquisition phase at TEMP currency is fundamentally significant for demonstrating 
the breadth of data sets across numerous acquisition phases, dissimilar program types, 
and different ACAT levels. 

One effective method for comparing test burden is by associating the number of 
test events conducted or proposed in the program’s Event Design Plan (EDP) to another 
test parameter. However, the representative data on Army-led test events are available 
only through ATEC, via the ATEC Decision Support System (ADSS) website. Test 
event plans and results collected from ADSS do not thoroughly represent the actual total 
number of events since many programs utilized other services and private laboratories for 
T&E support, and these organizations were not enthusiastic in providing this type of 
program information. 

Originated in the CDD, previously called the Operational Requirements 
Document (ORD), Key Performance Parameters are a system’s minimum set of user- 
defined, functional and operational requirements necessary to fulfill the conditions and 
constraints stated in the Mission Needs Statement (MNS), and pilots the efforts of the 
System Development and Demonstration Phase. These KPPs must be approved prior to 
entrance into the SDD Phase, and “may be refined, with the approval of the requirements 
authority, as conditions warrant.” [15, p. 3.7.2.3] KPPs are a critical subset of the 
performance parameters, are included in the perfonnance portion of the APB, and 
represent the most significant capabilities or characteristics for a given system. Failure to 
meet the minimum or threshold value of performance can be cause for program to be 
reevaluated or tenninated. The total number of KPPs for each program is listed in Table 1. 

C. HAPTER SUMMARY 

The primary focus of this chapter was to define, present, and initially investigate 
all of the collected data, and determine whether principal and secondary data is beneficial 
to the development of a mathematical approach, or basis for elimination in the data 
analysis for answering the questions presented in this JAP. Furthennore, selected 
programmatic and service-specific information concerning each program were collected 
for reference and to help bridge gaps in formulating an enhanced comprehensive analysis. 
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Moreover, the elementary process of organizing the data into practical categories exposed 
inconsistencies and inadequacies, such as funding, test expendables, special equipment, 
and manpower estimates that were especially difficult comparisons and provided little 
insight to the objectives herein. 



DATA PARAMETER 

TOTAL 
NO. KPP 

TOTAL 
NO. CTP 

TOTAL NO. 
QUANTIFIABLE CTP 

ACAT 

PROGRAM 





FCS 

7 

13 

6 

I 

NBCRV 

2 

25 

16 


STRYKER 

5 

19 

11 


JBPDS 

5 

14 

6 

II 

JCAD 

0 

14 

11 


JSLNBCRS 

0 

22 

9 


ACADA 

0 

17 

13 


ALSI-1 

4 

4 

3 


BIDS 

4 

18 

6 


CBPSS 

6 

24 

10 


FOX 

0 

9 

7 


JBAIDS 

3 

7 

5 


JBSDS 

4 

21 

14 


JPACE 

2 

20 

10 

III 

JSGPM 

5 

41 

16 


JSLIST-GLOVE 

8 

27 

16 


JSLSCAD 

6 

29 

21 


JSTDS 

5 

61 

30 


LYOSS 

0 

11 

4 


M45 

0 

18 

15 


M56 

0 

12 

12 


M6 

0 

7 

3 


SHIPBRD-CPS 

0 

11 

9 


Table 1. KPPs and Quantifiable CTPs. 
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DATA PARAMETER 

TOTAL NO. 
MOE/MOS 

TOTAL NO. 

QUANTIFIABLE MOE/MOS 

ACAT 

PROGRAM 




FCS 

35 

1 

I 

NBCRV 

38 

5 


STRYKER 

24 

6 


JBPDS 

29 

8 

II 

JCAD 

69 

34 


JSLNBCRS 

19 

4 


ACADA 

39 

26 


ALSI-1 

5 

3 


BIDS 

12 

6 


CBPSS 

14 

7 


FOX 

15 

12 


JBAIDS 

15 

10 


JBSDS 

12 

6 


JPACE 

44 

14 

III 

JSGPM 

55 

16 


JSLIST-GLOVE 

25 

13 


JSLSCAD 

17 

11 


JSTDS 

51 

19 


LYOSS 

21 

8 


M45 

13 

8 


M56 

27 

10 


M6 

27 

10 


SHIPBRD-CPS 

6 

5 


Table 2. Total Number of MOEs and MOSs. 
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DATA PARAMETER 

NO. 

COI 

NO. TEST 
ARTICLES 

NO. 

TEMP 

PAGES 

NO. TEMP 
PAGES SEC. 
I-Y 

ACAT 

PROGRAM 






FCS 

6 

NA 

482 

93 

I 

NBCRV 

13 

9 

78 

58 


STRYKER 

9 

NA 

134 

57 


JBPDS 

4 

102 

136 

67 

II 

JCAD 

2 

250 

160 

95 


JSLNBCRS 

3 

25 

74 

62 


ACADA 

3 

140 

54 

25 


ALSI-1 

4 

9 

71 

30 


BIDS 

3 

12 

44 

37 


CBPSS 

5 

18 

67 

43 


FOX 

3 

23 

30 

29 


JBAIDS 

2 

20 

110 

70 


JBSDS 

3 

14 

108 

40 


JPACE 

4 

390 

102 

75 

III 

JSGPM 

3 

5750 

84 

46 


JSLIST-GLOVE 

3 

678 

105 

51 


JSLSCAD 

4 

29 

76 

42 


JSTDS 

3 

250 

83 

61 


LVOSS 

2 

1772 

20 

16 


M45 

4 

800 

41 

35 


M56 

2 

18 

25 

17 


M6 

2 

127 

23 

18 


SHIPBRD-CPS 

2 

4 

27 

23 


Table 3. COIs and TEMP Page Counts. 
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ACAT 


I 


II 


III 


PROGRAM 

FCS 

NBCRV 

STRYKER 

JBPDS 

JCAD 

JSLNBCRS 

ACADA 

ALSI-1 

BIDS 

CBPSS 

FOX 

JBAIDS 

JBSDS 

JPACE 

JSGPM 

JSLIST-GLOVE 

JSLSCAD 

JSTDS 

LVOSS 

M45 

M56 

M6 

SHIPBRD-CPS 


Table 4. TEMP Data and 


DATA PA1 

RAMETER 

OTAL 

SPOWER 

TEMP 

DATE 

TEMP 

MILESTONE 

OSD/DOT&E 

OVERSIGHT 





NA 

25-Apr-2003 

MSB 

YES 

NA 

3-Sep-2004 

LRIP 

YES 

NA 

2-May-2003 

LRIP 

YES 

485 

15-Jun-2005 


YES 

NA 

1-Sep-2005 

MSIII 

YES 

238 

23-May-2003 

LRIP 

YES 

NA 

9-Dec-1997 

FRP 

NO 

NA 

30-0ct-2006 

SI-IPR/C 

NO 

NA 

12-Sep-2001 


NO 

NA 

7-Jun-2002 


NO 

NA 

Apr-1998 


NO 

34 

4-Aug-2006 


YES 

90 

28-Apr-2004 

C 

YES 

174 

11-Mar-2005 


NO 

984 

10-May-2004 

C 

NO 

334 

30-Mar-2005 


NO 

90 

3-Feb-2006 

FRP 

YES 

NA 

28-Feb-2005 


NO 

NA 

Oct-1998 


NO 

NA 

Feb-1994 


NO 

NA 

Jul-2005 


NO 

NA 

Sept-2000 


NO 

269 

23-Mar-1994 


NO 


ersight Requirements. 
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ACAT 


PROGRAM 


EVALUATION ORGANIZATIONS 




AEC 

AMSAA 

COMOPTEVFOR 

AFOTEC 

MCOTEA 


FCS 

X 





I 

NBCRV 

X 






STRYKER 

X 






JBPDS 

X 





II 

JCAD 

X 


X 




JSLNBCRS 

X 






ACADA 

X 






ALSI-1 

X 






BIDS 

X 






CBPSS 

X 






FOX 

X 






JBAIDS 




X 

X 


JBSDS 

X 



X 



JPACE 



X 

X 

X 

III 

JSGPM 

X 



X 

X 

JSLIST- 

GLOVE 

X 



X 

X 



JSLSCAD 

X 






JSTDS 

X 


X 

X 

X 


LVOSS 

X 






M45 

X 

X 





M56 

X 






M6 

X 






SHIPBRD-CPS 



X 




Table 5. Evaluation Organizations Utilized. 
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ACAT 

PROGRAM 

ARMY TESTING ORGANIZATION 

ATC 

DTC 

DPG 

EPG 

YPG 

WSMR 

CRTC 

ARL 


FCS 

X 

X 

X 

X 

X 

X 

X 

X 

I 

NBCRV 

X 


X 

X 

X 

X 

X 

X 


STRYKER 

X 

X 

X 

X 

X 

X 

X 



JBPDS 

X 


X 



X 



II 

JCAD 

X 

X 

X 


X 

X 

X 



JSLNBCRS 

X 


X 

X 

X 

X 




ACADA 

X 



X 

X 

X 

X 



ALSI-1 

X 


X 

X 

X 

X 




BIDS 

X 

X 

X 



X 




CBPSS 

X 

X 



X 

X 




FOX 

X 


X 



X 




JBAIDS 


X 

X 







JBSDS 

X 


X 



X 




JPACE 


X 







III 

JSGPM 

X 

X 

X 


X 

X 

X 


JSLIST- 

GLOYE 

X 

X 

X 








JSLSCAD 

X 

X 

X 


X 

X 




JSTDS 

X 


X 



X 




LVOSS 

X 


X 

X 

X 

X 

X 



M45 



X 


X 


X 

X 


M56 

X 


X 


X 

X 

X 



M6 

X 


X 


X 

X 

X 



SHIPBRD-CPS 










Table 6. Army Test Organizations Utilized. 
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ACAT 

PROGRAM 

ARMY TESTING 0 

►RGANIZATION 

ECBC 

RTTC 

TRTC 

Ft 

DRUM 

OTC 

FITG 

Ft. 

Huachuca 

I 

FCS 

X 


X 





NBCRV 



X 





STRYKER 



X 


X 



II 

JBPDS 








JCAD 

X 


X 




X 

JSLNBCRS 








III 

ACADA 

X 


X 





ALSI-1 

X 







BIDS 

X 







CBPSS 

X 



X 

X 

X 


FOX 

X 







JBAIDS 








JBSDS 








JPACE 

X 







JSGPM 

X 


X 





JSLIST- 

GLOYE 








JSLSCAD 








JSTDS 

X 







LVOSS 

X 

X 

X 





M45 

X 


X 


X 



M56 

X 


X 





M6 

X 

X 

X 





SHIPBRD-CPS 









Table 7. Additional Army Test Organizations. 
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ACAT 

PROGRAM 

N 

AVY TESTING OR< 

GANIZA1 

noN 

NRL 

NAVSEA 

NAVAIR 

NSWC 

Norfolk 

Quantico 

Mayport 

I 

FCS 








NBCRV 








STRYKER 








II 

JBPDS 


X 





X 

JCAD 



X 

X 

X 

X 


JSLNBCRS 








III 

ACADA 








ALSI-1 








BIDS 








CBPSS 








FOX 








JBAIDS 








JBSDS 








JPACE 



X 





JSGPM 








JSLIST- 

GLOYE 








JSLSCAD 








JSTDS 








LVOSS 








M45 








M56 

X 







M6 








SHIPBRD-CPS 









Table 8. Navy Test Organizations Utilized. 
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ACAT 

PROGRAM 

NAVY TESTING i 

ORGANIZATION 

NOMI 

NCTRF 

NSC 

NAWCAD 

SPAWAR 

I 

FCS 






NBCRV 






STRYKER 






II 

JBPDS 






JCAD 






JSLNBCRS 






III 

ACADA 






ALSI-1 






BIDS 






CBPSS 





X 

FOX 






JBAIDS 






JBSDS 






JPACE 

X 

X 

X 

X 


JSGPM 






JSLIST- 

GLOYE 






JSLSCAD 






JSTDS 






LVOSS 






M45 






M56 






M6 






SHIPBRD-CPS 







Table 9. Additional Navy Test Organizations. 
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ACAT 

PROGRAM 

AIR] 

FORCE TESTING ORGANIZATION 

McLellen 

Englin 

Dahlgren 

AFFTC 

Pope 

Hill 


FCS 







I 

NBCRV 








STRYKER 








JBPDS 

X 

X 





II 

JCAD 



X 

X 

X 

X 


JSLNBCRS 








ACADA 








ALSI-1 








BIDS 








CBPSS 



X 





FOX 


X 






JBAIDS 


X 






JBSDS 








JPACE 


X 





III 

JSGPM 


X 





JSLIST- 

GLOYE 



X 






JSLSCAD 








JSTDS 

X 







LVOSS 


X 






M45 








M56 








M6 








SHIPBRD-CPS 








Table 10. Air Force Test Organizations. 
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ACAT 

PROGRAM 

AIR FORCE TESTING ORGANIZATK 

ON 

Martin Electronics 

WPAFB 

Cherry Point 

AFMSA 

I 

FCS 





NBCRV 





STRYKER 





II 

JBPDS 





JCAD 





JSLNBCRS 





III 

ACADA 





ALSI-1 





BIDS 





CBPSS 





FOX 





JBAIDS 




X 

JBSDS 





JPACE 





JSGPM 





JSLIST- 

GLOYE 


X 

X 


JSLSCAD 





JSTDS 





LVOSS 





M45 





M56 





M6 

X 




SHIPBRD-CPS 






Table 11. Additional Air Force Test Organizations. 


51 

































ACAT 

PROGRAM 

PRIVATE TESTING ORGANIZA1 

[TONS 

Alion 

Labs 

Battelle 

Lincoln 

Labs/MIT 

Whyle 

Labs 

I 

FCS 





NBCRV 





STRYKER 





II 

JBPDS 

X 

X 

X 

X 

JCAD 





JSLNBCRS 




X 

III 

ACADA 





ALSI-1 





BIDS 





CBPSS 





FOX 





JBAIDS 





JBSDS 





JPACE 


X 



JSGPM 





JSLIST- 

GLOYE 


X 



JSLSCAD 





JSTDS 


X 



LVOSS 





M45 





M56 





M6 


X 


X 

SHIPBRD-CPS 






Table 12. Private Test Organizations Utilized. 


52 






































IV. DATA ANALYSIS 


A. INTRODUCTION 

This chapter analyzes the data presented in Chapter III to develop answers to the 
primary and secondary research questions. The primary research question is, “Is the Test 
and Evaluation (T&E) Level of Effort (LoE) proportional to the AC AT level?” The 
secondary questions are “How does program oversight by the Director, Operational Test 
and Evaluation or the Army drive the T&E LoE?” and “Are Chemical/Biological (C/B) 
programs a special case for T&E LoE?” 

The analysis is presented in three parts, data scope and limitations, analysis 
approach, and data analysis. Understanding the limitations and scope of the data set is 
critical to the development of a defendable analysis approach and to accurate 
interpretation of the results obtained. Similarly, the analysis must be perfonned within 
the context of the data set and the results obtained qualified based upon the data and 
analysis limitations. The analysis presented in this section is framed to ensure that the 
context of the baseline data is maintained. 

B. LIMITATIONS AND SCOPE 

The principal data source for this effort was the TEMP for each program. Other 
program documents, including perfonnance specifications, OSD budget and program 
oversight data, Single Acquisition Management Plan (SAMPs), APBs, and requirements 
documents were used to fill TEMP data gaps and to help answer the secondary research 
questions. 

During data collection efforts, it became clear that the variability in the level of 
detail and the types of information included in CBD TEMPs would also limit the scope of 
this analysis effort. Although these TEMPs generally included an Integrated Program 
Summary (IPS) and cost data, the level of detail and consistency of data program-to- 
program did not support a comparative analysis of test schedules or of program R&D, 
test, and production costs. A comparison of program schedules and budgets, taken from 

the OSD(C) website, also did not generate data sufficient to perfonn a test costs 
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comparison of CBD programs. This lack of test-specific data re-scoped the analysis to 
the technical and operational requirements and testing listed in each TEMP and to the 
documentation scope of each TEMP. 

Another limitation of the analysis effort was the difficulty in gaining access to 
non-CBD TEMPs. The team was only able to obtain copies of TEMPs for two Army 
programs outside of the CBDP, Stryker and Future Combat Systems (FCS). Both of 
these programs are ACAT I-D. Therefore, analysis in support of the secondary research 
question, “Are Chemical/Biological programs a special case for T&E LoE?” is 
constrained by this limited data set. 

C. APPROACH 

The answer to the primary research question was developed through an analysis 
of the system requirements, documentation scope, and test scope. The secondary 
research questions were answered by further dividing the existing data into smaller 
subsets, based on oversight and CBD program status. 

1. System Requirements 

The initial approach to data analysis was to perfonn a comparison of the system- 
level requirements in each TEMP to the scope of the resultant testing, sorted by ACAT. 
Requirements were sorted into two main categories, Operational Effectiveness (OE), and 
Operational Suitability (OS). Each requirement was further categorized to a functional 
area (e.g., operation, interoperability, environmental, mobility, primary technical 
performance characteristic (such as detection), and logistics). This analysis approach 
allowed a comparison of detection systems, but broke down for other types of CBD 
programs, such as vehicles, shelters, and individual protection due to the lack of 
consistent requirement categorization and detail levels in TEMPs from different 
functional areas. 

A mathematical approach was developed to normalize data across TEMPs after it 
was detennined that direct comparisons of OE and OS would not be possible. The 
change to a mathematical analysis enabled a direct comparison of programs without the 
need for a requirement-to-requirement crosswalk. This requirements analysis technique 
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is based on calculation of a program-specific quantified requirements ratio using the 
MOE, MOS, and CTPs, followed by an ACAT-based summarization. A similar ratio was 
also calculated for COIs. The analysis results were then reviewed to determine what, if 
any, ACAT-specific trends were apparent. 

2. Documentation Scope 

Per DA Pamphlet 73-1, “the target size of a TEMP should be approximately 30 
pages, including pages for figures, tables, matrices, and so forth. Although annexes and 
attachments are excluded from the 30-page limit, their size should be kept to a 
minimum.” [11, p. 15] Likewise, the Navy’s Operational Test Director’s Guide suggests, 
“a TEMP is generally limited to 30 pages.” [16, p. 5-3] Since the Army and Navy 
provides explicit guidance for TEMP page counts, the documentation scope analysis 
consists of tallying the sections I - V and total page counts for the TEMPS, then 
comparing the averages for each ACAT. Secondarily, the effects of test program 
oversight and a time-based analysis of the scope of ACAT III TEMPs will be 
investigated. 


3. Test Scope 

The scope of testing involves many factors; including test sites used, types of 
tests, funding used, and time and manpower required for testing. The analysis of test 
scope will focus on the involvement of test conduct and evaluation organizations and the 
types of tests each program listed (e.g., MIL-STD-810 environmental tests) to determine 
whether there are any ACAT level biases to test scope. 

D. DATA ANALYSIS 

1. System Requirements 

It was noticed during data collection that some of the requirements in each TEMP 
were more technically-focused, or quantifiable, than others. The quantifiable versus non- 
quantifiable nature of each requirement was captured as a secondary characteristic of the 
data. 
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The simple set of questions developed to categorize whether a requirement is 
quantifiable are, 

• Can a detennination of whether the system meets the requirement be made 
by simple observation or as an observable byproduct of a test for another 
characteristic? If so, the requirement is not quantifiable. 

• Does determination of whether the system meets the requirement require a 
formal nontrivial test? If so, the requirement is quantifiable. 

• Does the requirement include performance parameters that only require a 
yes/no or go/no-go answer? If so, the requirement is not quantifiable. 

Typical fonnats for quantifiable requirements are, 

• MOE, Probability of Detection (Pd) = 92% for point concentration 

• MOE, Accomplish OMS/MP following a HEMP event 

• MOS, MTBFA > 246 hr 

• MOS, Minimum of 1000 mean miles between critical vehicle failures 

• CTP, Collective pressure system with 2.5 IWG overpressure at 90 CFM 

• CTP, Meets EMI requirements of MIL-STD-461, test RSI03 
Non-quantifiable requirements are more general in nature, such as, 

• MOE, Able to be powered by rechargeable and non-rechargeable 
batteries 

• MOE, Shall be worn inside combat footwear 

• MOS, Includes a confidence checker 

• MOS, Crew size of not more than 4 crew members 

• CTP, HMMWV and Ml 13 mounted ability 

• CTP, Equipped with run-flat tires 

The total quantifiable requirements per TEMP are listed in Table 1 for CTPs and 
in Table 2, for the combination of MOE and MOS. Two factors contributed to the 
merging of MOE with MOS in the analysis (where MOS includes suitability and 
survivability); the lack of discrimination between MOE and MOS within many of the 
TEMPs and the operational focus of both requirements when compared to the technically 
oriented CTPs. 

Determination of whether a requirement was quantifiable or non-quantifiable 
neither directly allowed a meaningful comparison of programs, nor did it provide insight 
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into any ACAT-spccilic relationships. Since the number of quantifiable and total 
requirements in the form of MOE, MOS, and CTPs varied widely over the twenty three 
programs analyzed, a straightforward comparison based upon numbers of requirements 
did not yield any useful relationships. A comparison of individual requirements did not 
prove feasible, also due to the varied nature of system-specific requirements, even within 
the CBDP. 

Only with the development of a program-specific ratio of quantifiable to total 
requirements, was it possible to compare programs in an analytical manner. The ratio 
developed is called the CTP/MOE-MOS ratio, or more succinctly the C/M-M ratio. 

Calculation of the C/M-M ratio requires the preliminary calculation of the 
quantifiable MOE and MOS ratio, and the calculation of the quantifiable CTP ratio. 
Once these values are defined, the C/M-M ratio is calculated by dividing the quantifiable 
CTP ratio by the quantifiable MOE and MOS ratio. The formula is, 

Quantifiable CTP Ratio 

C/M-M Ratio =- 

Quantifiable MOE-MOS Ratio 


where, 

Number of Quantifiable CTPs 

Quantifiable CTP Ratio =- 

Total Number of CTPs 


Quantifiable MOE+MOS 

Quantifiable MOE-MOS Ratio =- 

Total Number of MOE+MOS 


The program-specific C/M-M ratios are listed in Table 13, Calculated System 
Requirements Ratios, and graphed in Figure 7, Quantified C/M-M Ratio for All 
Programs. Table 13 also includes a COI-based ratio, which is defined as the number of 
COIs divided by the C/M-M ratio. The formula for the COI/C/M-M ratio is, 


57 



Number of COIs 


COI/C/M-M Ratio =- 

C/M-M Ratio 


ACAT 

PROGRAM 

Program System Req 

luirements Ratios 

Percent 

Quantifiable 

MOE/MOS 

Percent 

Quantifiable 

CTPs 

C/M-M 

Ratio 

COI/ 

C/M-M 

Ratio 

I 

FCS 

2.9 

46.2 

16.2 

0.37 

NBCRV 

13.2 

64.0 

4.9 

2.67 

STRYKER 

25.0 

57.9 

2.3 

3.89 

II 

JBPDS 

27.6 

42.9 

1.6 

2.57 

JCAD 

49.3 

78.6 

1.6 

1.88 

JSLNBCRS 

21.1 

40.9 

1.9 

1.54 

III 

ACADA 

74.3 

76.5 

1.0 

2.91 

ALSI-1 

60.0 

75.0 

1.3 

3.20 

BIDS 

50.0 

33.3 

0.7 

4.50 

CBPSS 

50.0 

41.7 

0.8 

6.00 

FOX 

80.0 

77.8 

1.0 

3.09 

JBAIDS 

66.7 

71.4 

1.1 

1.87 

JBSDS 

50.0 

66.7 

1.3 

2.25 

JPACE 

31.8 

50.0 

1.6 

2.55 

JSGPM 

29.1 

39.0 

1.3 

2.24 

JSLIST-GLOVE 

52.0 

59.3 

1.1 

2.63 

JSLSCAD 

64.7 

72.4 

1.1 

3.57 

JSTDS 

37.3 

49.2 

1.3 

2.27 

LYOSS 

38.1 

36.4 

1.0 

2.10 

M45 

61.5 

83.3 

1.4 

2.95 

M56 

37.0 

100.0 

2.7 

0.74 

M6 

37.0 

42.9 

1.2 

1.73 

SHIPBRD-CPS 

83.3 

81.8 

1.0 

2.04 


Table 13. Program-Level System Requirements Ratios 
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Both ratios were developed to provide a measure of the technical orientation of the 
operational requirements for a system. These ratios also introduce a level of abstraction 
to the analysis, which proved essential to perfonning meaningful comparisons of systems 


at the AC AT level. 



Figure 7. Quantified C/M-M Ratio for All Programs 



Figure 8. COI/C/M-M Ratios by ACAT 


59 

















Systems Requirements Ratio Summary 


ACAT 

Mean 

C/M-M 

Ratio 

Standard 

Deviation 

of 

C/M-M 

Ratio 

Mean + 
Standard 
Deviation 

Mean - 
Standard 
Deviation 

Standard 

Deviation 

as a 

Percent of 
C/M-M 
Ratio 
Mean 

Mean 

COI/ 

C/M-M 

Ratio 

I 

(AH) 

7.78 

7.36 

15.14 

0.41 

94.7 

2.31 

I 

(Stryker) 

3.59 

1.80 

5.39 

1.79 

50.2 

3.28 

II 

1.70 

0.21 

1.91 

1.48 

12.6 

2.00 

III 

(AH) 

1.22 

0.44 

1.66 

0.78 

35.9 

2.74 

III 

(Old) 

1.18 

0.63 

1.81 

0.55 

53.3 

2.90 


1.26 

0.26 

1.53 

1.00 

20.9 

2.57 


1.17 

0.14 

1.31 

1.04 

11.9 

2.56 


Table 14. System Requirements Ratio Summary by AC AT 
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C/M-M Mean - Variability by ACAT 



ACAT Levels 


Figure 9. Mean C/M-M Values by ACAT with Standard Deviations 
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After the requirements-based ratios were calculated at the program level, a 
summary at the ACAT level was performed. A review of the results for ACAT I, II, and 
III programs showed a high level of variability, as evidenced by the large standard 
deviations on the summarized data in Table 14, System Requirements Ratio Summary by 
ACAT. The standard deviation for all ACAT I programs (All Pgms) was 94.7 percent of 
the mean value for the C/M-M ratio and the standard deviation for ACAT III (all 
programs) was 35.9 percent of the mean value for the C/M-M ratio. A second look at the 
data showed that for ACAT I systems the FCS data was significantly outside of the 
Stryker vehicles data range, and for ACAT III programs whose TEMPs were dated more 
than five years ago, appeared to differ significantly from more recent TEMPs. Therefore, 
additional analysis was performed to look at what effects these data differences may have 
at the ACAT level. As a result of this finding, Table 14 includes multiple categories 
within ACAT I and ACAT III summarized data. Additionally, a category for ACAT III 
oversight programs was added to help answer the research question related to effects of 
oversight on programs. 

The calculations required to generate the data in Table 14 are, 

n 

£ [C/M-M Ratios] 

1 

Mean C/M-M ratio =- 

n 


where, 

n=number of ratios in the data set 


Standard Deviation (o) = 


j 2 ( * - x V 
V (ii-1) 


where, x = sample mean average, n = sample size 
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Mean + Cl = Mean C/M-M ratio + G of C/M-M ratios used to calculate the mean 
Mean - CJ = Mean C/M-M ratio - <7 of C/M-M ratios used to calculate the mean 


O as Percent of Mean = ( -—-] X 100 

\Mesm C/M-M Ratio ) 

n 

X [COI/C/M-M Ratios] 

1 

Mean COI/C/M-M ratio =- 

n 

where, n = number of ratios in the data set 

The C/M-M ratio data was also compared, on a frequency of occurrence basis, to 
determine if the ratios followed any discemable pattern, either overall or by ACAT level. 
Table 15, Frequency of C/M-M Value Occurrence by ACAT, lists each C/M-M value 
obtained versus the total number of occurrences per ACAT. This data is graphed in 
Figure 8, Frequency Distribution of C/M-M Values by ACAT. The figure shows data 
that appears to fall into a normal distribution with a bell-shaped curve for ACAT II and 
ACAT III programs. The broad range and limited data set, three data points, for ACAT I 
programs showed no discemable pattern in relation to the ACAT II and III programs. 
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C/M-M 

Ratio 

Frequency of C/M-M Value Occurrence 

ACAT I 

ACAT II 

ACAT III 
(Old) 

ACAT III 

(New) 

0.7 



1 


0.8 



1 


1.0 



4 


1.1 




3 

1.2 



1 


1.3 




4 

1.4 



1 


1.6 


2 


1 

1.9 


1 



2.3 

1 




2.7 



1 


4.9 

1 




16.2 

1 





Table 15. Frequency of C/M-M Value Occurrence by ACAT 
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The requirements analysis results have demonstrated that the C/M-M ratio does 
differentiate between programs based on ACAT. As a whole, the ACAT III programs 
have a lower average value for the C/M-M ratio. The ACAT II programs fit into the high 
end of a normal distribution of frequency of C/M-M ratios when combined with the 
ACAT III programs. The C/M-M ratios for ACAT I programs are highly variable and are 
typically much higher than the values found for ACAT II and III programs. A 
comparison of the quantity of program COIs to the C/M-M ratio does not demonstrate 
any discernable difference between ACAT levels. 

2. Documentation Scope 

The second phase of analysis in the investigation of the test-related LoE for 
programs involves determining the documentation burden posed by the program TEMPs. 
The overall size of each TEMP and the size of sections I to V was gathered and 
summarized at the ACAT level and then compared to identify any trends. The page 
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count data from Table 3, COIs and TEMP Page Counts, and the date and oversight status 
from Table 4, TEMP Data and Oversight Requirements, are summarized in Table 16, 
TEMP Documentation Size Summary by ACAT. The standard deviation of the TEMP 
Sections I-V page counts was also calculated and listed in Table 16. The FCS and 
Stryker ACAT I programs are listed separately in Table 16 due to the extreme differences 
in documentation scope for these vehicle-based programs. Finally, Table 16 presents the 
relationship between the mean page counts and the DA Pamphlet 73-1 recommendation 
of a 30-page maximum for TEMP Sections I-V. The data presented in Table 16 is 
graphed in Figure 11, TEMP Page Count Comparison by ACAT. 


TEMP Page Count Comparison 



ACAT I-FCS ACAT I - Stryker ACAT II ACAT III ACAT III (Old) ACAT III (New) ACAT III 

(Oversight) 


□ SECTIONS I-V JTOTAL 


Figure 11. TEMP Page Count Comparison by ACAT 
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ACAT 

TEMP Documentation Size Summary 

TEMP 
Section 
I-V No. 

Pages 

Standard 
Deviation 
of Mean 
TEMP 

Sec. I-V No. 
Pages 

Standard 
Deviation, 
as a Percent 
of TEMP 
Sec. I-V 
Mean No. 
Pages 

TEMP 

Sec. I-V 
Pages, as a 
Percent 
of 30 page 
DA PAM 
Guidance 

Mean 

TEMP Total 
No. Pages 

I 

(FCS)* 

93 

N/A 

N/A 

310% 

482 

I 

(Stryker)+ 

58 

0.71 

1.2% 

192% 

106 

II 

75 

17.8 

23.8% 

249% 

123 

III 

(AH) 

39 

17.8 

45.8% 

129% 

64 

III 

(Old) 

27 

9.7 

35.7% 

90% 

38 


52 

15.6 

30.0% 

173% 

90 


51 

16.8 

33.1% 

169% 

92 


Table 16. TEMP Documentation Size Summary by ACAT 


* (FCS data is from a single program TEMP, + Stryker data is from two program 
TEMPs) 

Multiple findings are present in the results of the page count analysis. First, only 
the older ACAT III program group met the 30-page recommendation for TEMP Sections 
I-V. Second, the TEMP documentation size for new ACAT III programs does not appear 
to depend upon program oversight status. Third, the documentation size for ACAT III 
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programs is comparable to that of the AC AT I Stryker programs. Fourth, the AC AT II 
programs’ TEMP documentation size was higher on average than all other programs, 
with one exception, the ACAT I FCS program. 

3. Test Scope 

The final phase of analysis in the investigation of the test-related LoE for 
programs involves reviewing the scope of test related efforts documented in the program 
TEMPs. The data contained in Tables 5 through 12 was analyzed to determine whether a 
test-specific relationship exists between a program’s ACAT, joint designation, or 
oversight status. The number of occurrences of each item was summed over the data 
category (e.g., ACAT II programs; which include the JBPDS, JCAD, and JSLNBCRS) 
then the mean calculated by dividing the total occurrences by the number of programs in 
the data set (e.g., three) to determine the value. 

Table 17, Test Scope Summary by ACAT, shows the results using the same 
ACAT-based categories as the requirements and documentation analysis sections. Table 
18, Test Scope Summary by Joint and Oversight Status, presents an ACAT neutral view 
of how a program’s jointness or T&E oversight status may impact the LoE. Both tables 
include columns for the number of programs in each category, the mean number of test 
evaluation organizations used, the mean number of Army and other test organizations 
used, and the total of test organizations. Table 17 is organized using the same ACAT- 
based breakout as in the requirements and documentation analysis to enable one-to-one 
comparisons of the results. Table 18 is organized to provide insight into joint versus 
Army-only programs and T&E oversight to see if these attributes drive test efforts. 

The test scope evaluation, while limited to a comparison of the numbers of 
organizations involved in test evaluation and test conduct, provides some insight into the 
effects of ACAT level, joint designation, and oversight status. A review of the number of 
test evaluation organizations involved in programs, as graphed in Figure 12 from data in 
column 3 of Tables 17 and 18, shows several relationships. 
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ACAT 

Test Scope Summary -1 

Number 

of 

Programs 

Mean 

No. of 
Evaluation 
Orgs. ( 

Mean No. of 
Army Test 
Orgs. 

Mean No. of 
Other Service + 
Commercial Test 
Orgs. 

Mean Total 
of all 

Test Orgs. 

I 

3 

1.0 

9.0 

0 

9.0 

II 

3 

1.3 

5.7 

5.7 

11.4 

III 

(All) 

17 

1.9 

5.2 

1.4 

6.6 

III 

(Old) 

9 

1.1 

6.1 

0.9 

7.0 


8 

2.4 

4.1 

2.0 

6.1 

III 

(Oversight) 

3 

1.7 

3.3 

0.7 

4.0 


Table 17. Test Scope Summary by ACAT 


ACAT 

Independent 

Program 

Type 

Test Scope Summary - II I 

Number 

of 

Programs 

Mean 

No. of 
Evaluation 
Orgs. 

Mean No. 
of Army 
Test Orgs. 

Mean No. of 
Other Service 
+ Commercial 
Test Orgs. 

Mean 
Total of 
all 

Test 

Orgs. 

Joint 

10 

2.2 

4.4 

1.9 

6.3 

Army-Only 

13 

1.1 

6.8 

0.6 

7.4 

Oversight 

9 

1.3 

6.0 

2.1 

8.1 

Non-Oversight 

14 

1.7 

5.6 

1.6 

7.2 


Table 18. Test Scope Summary by Joint and Oversight Designation 
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Number of Organizations/Sites 


Test Evaluation Organizations 


ACAT Comparison Non-ACAT Comparison 



Figure 12. Test Evaluation Organization Involvement 


T&E Organizations Used 


ACAT Comparison Non-ACAT Comparison 



(Old) (New) (Oversight) Designated Oversight 


| B Army Test Orgs. □ Other Test Orgs. | 


Figure 13. Number of Test Organizations/Sites Used 
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ACAT I, ACAT III (Old), and Army-Only programs utilized, on average, only 
one evaluation organization. This result makes intuitive sense, as all three of these 
program types are Army-only or non-joint designated programs. For these programs 
AEC filled the test evaluation role. 

ACAT III (New) and Joint Designated programs have the highest average number 
of involved evaluation organizations. It also was expected that these two categories of 
programs would have similar values, since seven of the eight ACAT III (New) programs 
evaluated were joint programs. The other joint programs analyzed were the three ACAT 
II programs, which had a lower value than the ACAT Ill’s, which pulled the joint 
designated average down slightly. This category had more than twice the average 
number of evaluators as the Army-only and ACAT I programs. 

ACAT III (Oversight) and all non-oversight programs had the same value of 1.7 
organizations, and ACAT II and all oversight programs had a lesser, but identical value 
of 1.3 organizations. These results demonstrate that program oversight generally reduces 
the number of test evaluators involved in a program, and that ACAT III oversight 
programs have a similar evaluator involvement as most programs without oversight. 

Figure 12 is split, with the left side showing an ACAT-based comparison, and the 
right side showing the non-ACAT factors that may contribute to differences between 
programs. The analysis of test evaluation organizations depicted in Figure 12 shows 
nearly identical means and standard deviations, regardless of the basis of comparison. 
The means and standard deviations are 1.57 and 0.54 for ACAT-based, and 1.58 and 0.49 
for non-ACAT-based views. Therefore, the ACAT and the other factors evaluated are 
equivalent in their influence on the results. 

The scope of T&E efforts related to numbers of test organizations, and test sites, 
used is depicted in Figure 13, which is also split to allow ACAT and non-ACAT 
comparisons. Most programs fall within a one standard deviation band of total test 
organization and site usage. As would be expected, Army-only programs tended to 
primarily use Army test sites. Only ACAT II and ACAT III (Oversight) programs fell 
outside of the one standard deviation band. ACAT II programs used as many non-Army 

test organizations and sites as Anny ones, while ACAT III (Oversight) programs are 
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significantly below this band. Interestingly, the category that used the fewest 
organizations and sites was AC AT III (Oversight). 

The non-AC AT comparison of test sites in Figure 13 shows a much lower 
standard deviation than the ACAT-based results. The means and standard deviations are 
7.35 and 2.55 for ACAT-based, and 7.25 and 0.74 for non-ACAT-based views. This 
demonstrates that the ACAT is the primary factor for differences in the number of test 
organizations and sites used. 

E. CHAPTER SUMMARY 

The analysis of the data shown in Chapter III was performed in three parts, 
system requirements, documentation scope, and test scope. The analysis was performed 
on twenty-three programs, primarily using data extracted from TEMPs, to answer the 
primary and secondary research questions. 

The requirements analysis has shown that ACAT III CB programs have the most 
technically-oriented, quantifiable, MOE and MOS. ACAT II CB program MOE and 
MOS are only slightly less quantifiable than those of ACAT III CB programs. The 
ACAT I programs evaluated had the least technically focused MOE and MOS 
requirements. 

The documentation scope analysis determined that none of the current CB and 
Army programs met the DA Pamphlet 73-1 guidance for Sections I-V TEMP pages. 
Also, TEMPs for an average ACAT III CB program are equivalent in volume to the 
ACAT I Stryker programs. The impact of oversight could only be evaluated for ACAT 
III programs, where no noticeable differences in TEMP scope could be discerned. 

The test-related analysis determined that the average number of test evaluation 
organizations used was not directly related to the ACAT level. There were two outliers 
in the analysis, one was ACAT I programs and the other was joint-designated programs. 
ACAT I programs used only one evaluation organization, while joint-designated 
programs averaged more than twice that number. 

The test related analysis also did not find any noticeable trends between for the 
joint designated, Army-only, oversight, or non-oversight categories. However, ACAT III 

(Oversight) programs and ACAT II programs were statistically outside of the norm for 
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average number of test organizations and sites used. The extensive use of non-Army test 
facilities drove the ACAT II programs significantly higher than other program categories, 
while the reason that ACAT III (Oversight) programs were significantly lower than other 
programs was not directly evident. 

In summary, the analysis has detennined that, 

• ACAT II programs are the most burdened CB programs; combining 
highly-technical MOEs and MOSs, joint-designation, the highest TEMP 
documentation burden, and the broadest test scope. 

• ACAT I programs are the least burdened by technical MOE and MOS 
requirements, while ACAT III programs are the most burdened. 

• Oversight appears to have a neutral to positive impact on ACAT III 
programs. 

• The TEMP documentation burden for CB programs is not related to 
ACAT. 

• Joint-designation noticeably increases the test and documentation burden. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

This chapter presents the conclusions reached regarding the primary and 
secondary research questions of this study. The conclusions are derived from the analysis 
conducted in Chapter IV, using the background information presented in Chapter II, the 
data presented in Chapter III, and the overall scope of the study. Recommendations for 
further research are presented in order to focus potential follow-on investigations toward 
areas that will help complete the overall picture of ACAT-related program test burden. A 
recommendation related to TEMP implementation is also presented to foster discussions 
of T&E management processes. 

B. CONCLUSIONS 

Level of Effort is a key phrase in all three study research questions. This research 
study evaluated the LoE for T&E within a program using three factors, 

• How technically focused the operationally evaluation parameters, MOE 
and MOS, were in relation to the technical evaluation parameters, CTPs. 

• The T&E documentation burden to a program, based on the TEMP size. 

• The number of T&E organizations involved in the conduct and evaluation 
of program tests. 

1. Primary Research Question - T&E LoE and ACAT 

The primary research question is, “Is the Test and Evaluation (T&E) Level of 
Effort (LoE) proportional to the ACAT level?” 

The T&E LoE is not proportional to the ACAT level. 

Each ACAT level scored best in one or more, and worst in one or more, of the 
categories evaluated. The T&E LoE was analyzed using the three factors listed above. 
The ACAT I programs evaluated had the best differentiated technical and operational 
requirements and used the least number of evaluation organizations. ACAT II programs 
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had technically-based operational requirements, a high LoE for TEMP documentation, 
and used the most test sites. ACAT III programs had the most technically-based 
operational requirements, similar TEMP documentation to the other ACATs, with the 
most evaluator involvement, but used less test sites overall. These conflicting results 
cemented the conclusion that ACAT level and T&E LoE do not correlate directly. 

2. Secondary Research Question - Oversight 

The first secondary research question is “How does program oversight by the 
Director Operational Test and Evaluation or the Army drive the T&E LoE?” 

T&E oversight had a neutral to positive effect on ACAT III CB programs. 

Oversight comparisons could only be perfonned for ACAT III programs, as there 
were no non-oversight ACAT I and II programs in this study. ACAT III oversight 
programs showed improvements over non-oversight programs by using fewer test sites 
and through less variability in the C/M-M ratio within the TEMP. The overall C/M-M 
ratio, the TEMP LoE, and the number of test evaluators were all comparable to non¬ 
oversight ACAT III programs. 

3. Secondary Research Question - CB Programs 

The second secondary research question is “Are Chemical/Biological (C/B) 
programs a special case for T&E LoE? 

An answer to this question could not directly be determined. 

Since only three of the programs studied were not part of the CBDP, and they 
were all ACAT I programs, there was insufficient data to detennine whether CB 
programs are unique within DoD for their T&E LoE. Data related to the joint nature of 
most CB programs was gathered and analyzed, but this data was not broad enough to 
answer this overarching question. 
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C. RECOMMENDATIONS 

1. Areas of Further Study 

Although this research study was able to find useful relationships between 
programs of different ACATs, including the development of a new requirements 
quantification ratio, it was constrained by the limitations of the available data. Specific 
additional study areas that have the potential to significantly increase the overall 
understanding of how T&E is performed on different ACAT-level programs are, 

• An analysis of acquisition programs in other commodity areas within 
DoD, using this study’s methodology. This research will help detennine 
whether CB programs are typical or unique within DoD. 

• An analysis of program T&E funding to overall program funding on an 
AC AT and non-AC AT basis, for both Research, Development, Test & 
Evaluation (RDT&E) and Procurement funds for CB programs. This 
research will enable a requirements versus cost analysis, which was not 
able to be completed with existing TEMP cost data. 

• An analysis of the number and types of tests conducted, based upon 
ACAT and non-ACAT factors. This information will provide the final 
details necessary to determine what, if any, factors drive detailed program 
tests. 

2. T&E Program Implementation 

One area that should be reevaluated is the Army’s recommendation for no more 
than 30 pages for Sections I-V of a TEMP. Only the older ACAT III programs studied 
met this goal, and current Army/CBDP TEMPs appear to have normalized on a 50-60 
page average. Since none of the current acquisition programs studied met this 
requirement, regardless of ACAT, the time may be right to rethink this guidance. 

D. SUMMARY 

This research study was undertaken to investigate the validity of the common 
belief that smaller programs endure a disproportionately heavy T&E burden relative to 
larger programs. In general, this notion was proven true. ACAT III programs have the 
most technically detailed operational requirements, but do gain some benefit from 
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focused oversight. However, AC AT II programs endure the ‘worst of all worlds’ when it 
comes to T&E burden. It appears that AC AT II CB programs are managed as if they 
were AC AT I-D OSD/DOT&E oversight programs, while also having an AC AT Ill-like 
burden of highly technically-biased operational requirements. Further study is needed to 
determine whether this T&E bias applies to non-CB programs, or is unique to the CBDP. 
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APPENDIX. PROGRAM FACT SHEETS 


Future Combat Systems (FCS) 

The Army's Future Combat Systems (FCS) network allows the FCS Family-of-Systems (FoS) to 
operate as a cohesive system-of-systems where the whole of its capabilities is greater than the 
sum of its parts. As the key to the Army's transformation, the network, and its logistics and 
Embedded Training (ET) systems, enable the Future Force to employ revolutionary operational 
and organizational concepts. The network enables Soldiers to perceive, comprehend, shape, and 
dominate the future battlefield at unprecedented levels as defined by the FCS Operational 
Requirements Document (ORD). 

The FCS network consists of four overarching building blocks, System-of-Systems Common 
Operating Environment (SOSCOE); Battle Command (BC) software; communications and 
computers (CC); and intelligence, reconnaissance and surveillance (ISR) systems. The four 
building blocks synergistically interact enabling the Future Force to see first, understand first, act 
first and finish decisively. 

http, //www.army.mil/fcs/overview.html (Nov 2007) 



http, //www4.army.mil/OCPA/uploads/large/FCSnetwork2004-07-23.jpg (Nov 2007) 
http, //www4.army.mil/OCPA/uploads/large/Mortar2004-l 0-19.jpg (Nov 2007) 
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NBC Reconnaissance Vehicle (NBCRV) - Stryker Variant 


The Nuclear, Biological, Chemical Reconnaissance Vehicle (NBCRV) provides on the move and 
remote near-real-time nuclear, biological and chemical detection and surveillance to supply 
battlefield visualization of NBC hazards. 

General, 

The NBC RV provides NBC situational awareness to increase the combat power of the SBCT. 
The core of the NBC RV is its on-board integrated NBC sensor suite and integrated 
meteorological system. An NBC positive overpressure system that minimizes cross¬ 
contamination of samples and detection instruments, provides crew protection, and allows 
extended operations at MOPP 0. 

Operational Capability, 

The NBCRV will have the capability to detect and collect chemical and biological contamination 
in its immediate environment on the move through point detection (CBMS and JBPDS), and at a 
distance through the use of a stand off detector (JSLSCAD). It automatically integrates 
contamination information from detectors with input from on-board navigation and 
meteorological systems and automatically transmits digital NBC warning messages through the 
Maneuver Control System to warn follow-on forces. Mission, NBC Reconnaissance. Find, 
identify, map, and mark NBC contamination on the non-linear battlefield. 

• 32-57% reduced route reconnaissance mission time via on-the-move standoff chemical 
agent detection 

• Common NBC technical architecture 

• Biological Detection Capability (CBMS and JBPDS) 

Digitized division/corps-fused NBCRS architecture 

Stryker NBCRV Technology 



LAPTOPS 


LASER | 
PRINTER 


DWSS 


AN/UDR- 
13 Radiac 


AN/VDR-2 

Radiac 


ACADA CBMS II 


METSMAN 


JSLSCAD 


http, //www.sbct.army.mil/ (Nov 2007), http, //www .jpeocbd.osd.mil/Recon/Products-CA- 
NBCStryker.html (Nov 2007) 
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Stryker Family of Vehicles 


Stryker™ is a family of eight-wheel drive combat vehicles, transportable in a C-130 aircraft, 
being built for the US Army by GM GDLS, a joint venture set up by General Motors Defense of 
Canada and General Dynamics Land Systems Division of USA. Stryker is based on the GM LAV 
III 8 x 8 light-armored vehicle, in service since early 2001. The LAV III is itself a version of the 
Piranha III built by Mowag of Switzerland, now owned by General Motors Defense. GM Defense 
and GDLS are sharing the fabrication and final assembly of the vehicles among plants at 
Anniston, Alabama; Lima, Ohio; and London, Ontario. 

The Stryker Brigade Combat Team (SBCT) combines the capacity for rapid deployment with 
survivability and tactical mobility. The Stryker vehicle enables the team to maneuver in close and 
urban terrain, provide protection in open terrain and transport infantry quickly to critical 
battlefield positions. 

GM Defense and GDLS were awarded the contract for the US Army's Interim Armored Vehicle 
(IAV) in November 2000. 

The eight-wheeled Stryker is the first new military vehicle to enter service into the United States 
Army since the Abrams tank in the 1980s. 

The United States Army first deployed 14 Stryker vehicles as part of its forced entry package for 
Millennium Challenge 2002, the Joint Forces Command field experiment and demonstration in 
July and August 2002. The Strykers were deployed from C-130 and C-17 aircraft during the 
exercise. Formal brigade certification is planned for May 2003. 

Variants, 

Stryker variants include the Infantry Carrier Vehicle (ICV) and the Mobile Gun System (MGS). 
There are eight configurations of the ICV including Nuclear, Biological, Chemical 
Reconnaissance Vehicle (NBC RV); Anti-Tank Guided Missile (ATGM); Medical Evacuation 
Vehicle (MEV); Mortar Carrier (MC); Engineer Squad Vehicle (ESV); Commander's Vehicle 
(CV); Fire Support Vehicle (FSV); and the Reconnaissance Vehicle (RV). They have parts 
commonality and self-recovery abilities and are equipped with a central tire inflation system. 



http, //www.sbct.army.mil/ (Nov 2007) 
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Joint Biological Point Detection System (JBPDS) 

Description, 

The Joint Biological Point Detection System (JBPDS) Acquisition Category II (ACAT II) 
Sentinel program is the successor to the Army BIDS, Navy IBAD, and the Air Force service 
specific development programs. The JBPDS will meet Quad-service requirements as outlined in 
the Joint Operational Requirements Document (JORD) and consist of complementary trigger, 
sampler, detector and identification technologies to rapidly and automatically detect and identify 
biological threat agents. The suite will be capable of identifying BW agents in less than 15 
minutes. The suite will be capable of identifying, as a minimum, BW agents listed in Category A 
of the International Task Force (ITF) 6 Report, dated Feb 90. The detection suite will be 
integrated into each Service's platform (e.g. BIDS, surface ships, LNBCRS) or air base and port 
to provide a common detection capability for joint interoperability and supportability. The JBPDS 
will increase the number of agents that can be identified by the BIDS and the IBADS; decrease 
detection and identification time; increase detection sensitivity; provide automated knowledge- 
based detection and identification; and provide a first time point detection capability to the Air 
Force and Marine Corps. 

Mission, 

To detect, identify, sample, collect and 

communicate the presence of biological warfare 

agents to enhance the survivability of U.S. Forces. 

Capabilities, 

- Point detection capability for all Services 

- Increased reliability and maintainability 

- Identify 10 biological warfare agents 
simultaneously 

Improvements over the BIDS and IBAD, 

- Lower probability of false positive identification 

- Interface with JWARN System and FBCB2 

- Fully automated detection and identification 
capability 

User, 

U.S. Army, U.S. Navy, U.S. Air Force, and U.S. 

Marine Corps 



http, //www.jpeocbd.osd.mil/page_manager.asp?pg=7&sub=18 (Nov 2007) 
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Joint Chemical Agent Detector (JCAD) 

Description, 

The JCAD will be a combined portable monitoring and small point chemical agent detector for 
aircraft, shipboard, and individual soldier applications. This hand-held, pocket-sized detector is 
required to automatically detect, identify, and quantify chemical agents inside the aircraft or ship, 
providing protection for the individual Soldier, Sailor, Airman, or Marine. For the duration of the 
mission, the device must be sufficiently sensitive to warn aircrews before accumulation of a dose 
that will cause miosis or more severe effects. It must be resistant to the severe interferent 
environment on a naval vessel and be small and rugged for individual use. 

Mission, 

Advanced detection and warning, identification of contamination on personnel and equipment, 
and monitoring for the presence of chemical warfare agent contamination 

Capabilities, 

- Instant feedback of hazard (mask only or full MOPP) 

- Real-time detection of nerve, blister, and blood agents 

- Miosis-level detection capability (with pre-concentrator) 

- Calculates accumulated dosage 

- Stores up to 72 hours of detection data 

- Fully compatible with Joint Warning and Reporting System (JWARN) 

Users, 

U.S. Air Force (Program Lead), U.S. Army, U.S. Navy, and U.S. Marine Corps Replaces the 



http, //www.jpeocbd.osd.mil/Point/Products-CA-JCAD.html (Nov 2007) 
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Joint Services Lightweight NBC Reconnaissance System (JSLNBCRS) 


Description, 

The JSLNBCRS will provide point and standoff intelligence for real-time field assessment of 
NBC hazards. The system is a vehicle-mounted suite of equipment and software designed to 
detect, collect, analyze, mark, and disseminate NBC data. Two variants, the High Mobility 
Multipurpose Wheeled Vehicle and the Light Armored Vehicle, will house the same equipment 
and offer on-the-move, standoff capability, while providing an air-transportable system. Timely 
provision of automated, digital information meshed with meteorological and positioning 
information will provide commanders more options in merging NBC information with tactical, 
operational, and strategic plans. 

Mission, 

Perform NBC Reconnaissance. The JSLNBCRS will detect, identify, mark, collect, correlate, and 
disseminate NBC hazard and toxic industrial chemical information over an automated warning 
and reporting information network. 

Capabilities, 

- JSLSCAD - On-the-Move near real-time chemical vapor detection 

- CBMS II- Chemical Detection 

- JBPDS- Biological Detection 

- ACAD A - Chemical warfare agent detection (nerve and blister agents) 

- ADM 300A - AN/VDR-2 - Measures radioactivity for either vehicle-mounted or hand-held 
operations 

- ICAM/CAM II - Hand-held monitoring of chemical agents 

- JWARN to analyze and report NBC hazards 
Improvements, 

- On-the-move standoff chemical detection 

- Biological detection and sampling 

- On-the-move meteorological system 

- Significantly less impact on strategic and intra-theater lift resources 

- Collective protection for crew against known CBW agents. 

- Generation of automated NBC warning messages. 


Users, 

U.S. Marine Corps and U.S. Air Force 



http, //www.jpeocbd.osd.mil/Recon/Products-CA-JSLNBCRS.html (Nov 2007) 
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M22/M88 Automatic Chemical Agent Detection Alarm (ACADA) 


Description, 

The M22 Automatic Chemical Agent Alarm (ACADA) is an automatic chemical agent alarm 
system capable of detecting, warning and identifying standard blister and nerve agents 
simultaneously. The M22 is man-portable, operates independently after system start-up, provides 
an audible and visual alarm and provides communication interface to support battlefield 
automation systems. 

Mission, 

Provide detection and warning for nerve and blister agents. 

Capabilities, 

- Area warning 

- Collective Protection Equipment (CPE) monitoring 

- Operation on and in vehicles 

- Compatible with MICAD 

- Provides simultaneous detection and warning of nerve and blister agents 

- Significantly more sensitive than M8A1 

- Operates in a collective protection environment 

- Much less responsive to interferences 

- Able to operate in and on vehicles 

Users, 

U.S. Army, U.S. Navy, U.S. Air Force and U.S. Marine Corps 



http, //www.jpeocbd.osd.mil/Point/Products-CA-ACADA.html (Nov 2007) 
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Analytical Laboratory System (ALS) 


Description, The Analytical Laboratory System (ALS) is a C-130 air transportable system 
thatuses commercial-off-the-shelf (COTS) equipment that can analyze ChemicalWarfare (CW) 
agents, Toxic Industrial Materials (TIM), Toxic Industrial Chemicals (TIC) and Biological 
Warfare (BW) agents. The ALS has the capability of establishing communications through the 
Unified Command Suite (UCS) to local, state, federal laboratories and other Agencies for 
confirmatory analysis of supect agent. 

Mission, The ALS provides the capability to the Civil Support Teams to conduct presumptive 
analysis of u nk nown or potential agents at an incident site and transmit that information 
electronically via the UCS in support of the First Responder Incident Commander. 

Capabilities, 

Detects and identifies GB, GD and HD 

Self - contained and fully functional and reliable in the range of extreme climates encountered in 
the United States and its territories 

Provides results from screening of TIC/TIM/CW samples within 30 minutes 
Provides results from screening of BW samples within 45 minutes 

User, National Guard Bureau, Weapons of Mass Destruction - Civil Support Teams. 
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Biological Integrated Detection System (BIDS) 

Description, 

The Biological Integrated Detection System (BIDS) is an early warning and identification 
capability in response to a large area (theater) Biological Warfare (BW) attack. The system is a 
detection suite installed in a shelter that is mounted on a dedicated vehicle with trailer mounted 
generator power supply. Other BIDS elements include collective protection, environmental 
control, sample storage, GPS, MET, and radios. The BIDS pre-planned product improvement 
BIDS (P3I) system is equipped with a detection suite to include high volume samplers, a 
fluorescent particle counter/sizer, a flow cytometer, a chemical/biological mass spectrometer and 
an antibody-based biological detector. The shelter may be removed from the vehicle for fixed site 
application. The BIDS program was conducted in three phases. Phase I was the non- 
developmental item (NDI) BIDS. Phase II was the P3I, which provided technology insertion to 
upgrade the NDI core configuration from a four-agent to an eight-agent identification capability 
with improved generic detection. The objective system (Phase II) transitioned to JBPDS. 

Mission, 

Detect and identify large-area Biological Warfare (BW) agent attacks, provide a basis for large- 
area protection and warning. 

Capabilities, 

- BIDS NDI 

- Detect/Identify 25 ACPLA/30Min 

- Manual operation 

- BIDS P3I 

- Detect/Identify 15 ACPLA/20Min 

- Semi-automatic operation 

- BIDS Obj System 

- JBPDS Bio Suite 

Improvements over prior systems, 

- First Army biological detection capability 

- Basis for rapid detect-to-treat decisions 

- Provide improved situational awareness 

User, 

U.S. Army 
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Chemical and Biological Protective Shelter (CBPS) 


Description, 

The Chemical and Biological Protective Shelter (CBPS) provides a highly mobile, contamination 
free, environmentally controlled work area for forward deployed medical treatment units. The 
CBPS consists of a dedicated Ml 113 HMMWV vehicle, a Lightweight Multipurpose Shelter 
(LMS) mounted on the back of the HMMWV, a 300 ft 2 semi-cylindrical airbeam-supported soft 
shelter, and a towed High Mobility Trailer with lOkW Tactical Quiet Generator (TQG) for 
auxiliary power. Four soldiers operate the CBPS; during transport, two ride in the HMMWV and 
two in the LMS. Soldier’s gear and required medical equipment will be stored in the LMS or on 
the trailer. The inflatable shelter is rolled and carried on the rear of the LMS during transport. All 
power required to support operations of the sys-tem is provided by the HMMWV engine via an 
underhood hydraulic pump, or from the auxiliary lOkW TQG. A hydraulically-powered 
Environmental Support System mounted on the front of the LMS provides heating, cooling, 
airbeam inflation, CB filtration, and ventilation air. 

Mission, 

CBPS will be fielded to Treatment Squads, Forward Surgical Teams, and Medical Companies 
and will allow medical personnel to work without encumbrances of protective clothing in a CB 
contaminated environment. 

User, Full Materiel Release achieved and fielded to U.S. Army and U.S. Marine Corps during 

Operation Iraqi Freedom 

Capabilities, 

• Highly Mobile self-contained system 

• CBPS can be fully operational in less than 20 minutes 

• Personnel work inside without encumbrances of protective gear 

• Provides a clean environment for medical care, to include a desert environment 

O O 

• Environmentally controlled sustained operations -25 F to 120 F 
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M93A1 Fox Nuclear, Biological, Chemical Reconnaissance System 
(NBCRS) 


Description, 

The M93A1 FOX NBCRS is a dedicated system of nuclear and chemical sampling, detection, and 
warning equipment, and biological sampling equipment integrated into a high speed, high 
mobility, six wheel armored vehicle. It is capable of performing nuclear, biological, and chemical 
(NBC) reconnaissance on primary, secondary, or cross-country routes throughout the battlefield. 
The M93A1 is the Block 1 improvement of the M93 FOX NBCRS Interim System to meet all of 
the requirements of the approved required operational characteristics (ROC), and reduce the crew 
size to three. It is capable of detecting chemical contamination in its immediate environment 
through point detection, and at a distance through the use of a standoff detector (M21 Chemical 
Agent Detector Alarm). It automatically integrates contamination information from detectors with 
input from on-board navigation and meteorological systems and automatically transmits digital 
NBC warning messages through the Maneuver Control System to warn follow-on forces. 

Mission, 

NBC Reconnaissance. Find, identify, map, and mark NBC contamination on the battlefield. 
Capabilities, 

- Standoff vapor detection 

- Reduced crew size 

- Digitized communications 

- On-board MICRO-MET, GPS, SINCGARS 

- Automatic Alarm Reporting (MICAD) 

Future Improvements, 

- FOX Recapitalization - Zero hours/Zero Miles, Enhanced Sensor Suite, Improved Survivability 
and Lethality 

- Significantly reduced operational, sustainment and life cycle costs 
Users, 

U.S. Army, U.S. Marine Corps 



http, //www.jpeocbd.osd.mil/Recon/Products-CA-FOXBlockI.html (Nov 2007) 


89 




Joint Biological Agent Identification and Diagnostic System (JBAIDS) 


Description, The Joint Biological Agent Identification and Diagnostic System (JBAIDS) is an 
integrated system for rapid identification and diagnostic confirmation of biological agent 
exposure or infection. Based on commercial technology, JBAIDS is man-portable, reusable, and 
will be capable of the simultaneous identification of multiple Biological Warfare agents (BWA) 
and other pathogens of operational concern. The system consists of the hardware platform to 
perform sample analysis, a laptop computer for readout display, and assay reagent test kits. 

Mission, 

Provide rapid positive identification and diagnostic confirmation of BWA and other pathogens of 
operational concern. 

Capabilities, 

• Single DoD accepted platform for both identification and diagnostic confirmation of 
biological agents 

• Operation in fixed medical laboratories and deployed medical units 

• Operates as a stand-alone system; future development Blocks II and III to be 
interoperable with Theatre Medical Information Program (TMIP) 

Improvements over existing technology, 

• Provides simultaneous identification of multiple biological agents 

• Rapid identification 
User, 

U.S. Army, U.S. Navy, U.S. Air Force, and U.S. Marine Corps. 
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Joint Biological Standoff Detection System (JBSDS) 


Description, 

The JBSDS is in development and is the first joint biological standoff detection program. The 
JBSDS will be a standoff early warning biological detection (BD) system. The system will be 
capable of providing near real time, detection of biological attacks/incidents and standoff early 
warning detection/warning of biological warfare (BW) agents at fixed sites or when mounted on 
multiple platforms, including NBC reconnaissance platforms. It will be capable of providing 
standoff detection, ranging, tracking, discrimination (manmade vs natural occurring aerosol) and 
generic detection (bio vs non-bio) of large area BW aerosol clouds for advanced warning, 
reporting, and protection. 

Mission, 

To provide a biological detection network capable of near real time detection and warning, 
theater wide, to limit the effects of biological agent hazards at the tactical and operational levels 
of war. 

Capabilities, 

- Provides integrated, stand-off detection near real time for Bio- Detect and track aerosol clouds 
out to 15km 

- Operate at fixed site or stationary mode from mobile platforms 

- Discriminate Bio particles from non Bio particles in aerosol clouds out to 3km 

- Operationally skin and eye safe 
Improvements over previous systems, 

- JBSDS will provide early warning BWA standoff for all Services. Currently fielded biological 
detection systems do not. 

- JBSDS will discriminate between natural and man-made aerosols. 

- JBSDS will not require aviation assets for employment. Existing LRBDS does. 

User, 

U.S. Army, U.S. Navy, U.S. Air Force, and U.S. Marine Corps 
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Joint Protective Aircrew Ensemble (JPACE) 


Description, 

The Joint Protective Aircrew Ensemble (JPACE) is a Chemical Biological Protective garment 
that is fire resistant and provides protection from Chemical/Biological (CB) warfare agents and 
radiological particles. JPACE is intended for use by all aviators and aircrew for fixed wing 
ejection, fixed wing non-ejection, and rotary wing personnel. 

Mission, 

Provide below-the-neck chemical/biological protection for aviators and aircrew personnel. 
Capabilities, 

♦ Chemical protection from all liquid, particle, vapor and aerosol CB agents 

♦ Provides protection to enhance combat capabilities in a CB environment 

♦ Compatible with all services aviation ALSS/ALSE, legacy Chem/Bio Aviation mask systems, 

all developmental masks, JSLIST Glove Block Elpgrades, foot wear solutions, and the 
Army’s Air Warrior system. 

Improvements over the current garments, 

Replaces, Navy Mkl, CPU-CWU-27P; 

Army ABDU-BDO, CPU-ABDU; Air 
Force CWU-66/PIncreased CWBW agent 
protection 

♦ Increased flame protection 

♦ Longer post exposure protection 

♦ Decreased heat stress 

Repair kits for both the outer shell and 
the line for field expedient and repair for 
reuse. 

Laundering cycles, shelf/service life is 
increased. 

User, 

U.S. Army, U.S. Navy, U.S. Air Force, 

U.S. Marine Corps, U.S. Coast Guard 
and U.S. Special Operations Command. 
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XM50/51 Joint Service General Purpose Mask (JSGPM) 


The JSGPM will be a lightweight, protective mask system (consisting of mask, carrier and 
accessories) incorporating state of the art technology to protect U.S. forces from anticipated 
threats. The mask components will be optimized to minimize impact on the wearer's performance 
and to maximize its ability to interface with future Service equipment and protective clothing. 

Mission, 

Provide face, eye, and respiratory protection from battlefield concentrations of Chemical and 
Biological (CB) agents, toxins, toxic industrial materials, and radioactive particulate matter. 

User, 

All Services - Replaces the M40/42 and MCU-2/P Series Masks, and the M45 in the Land 
Warrior Program. 

Target Capabilities, - Provide 24 hours of above the neck protection from CB agents, radioactive 
particles, and toxic industrial materials- Overall field of view 80% 

- Inhalation resistance 30 mm water 

- Compatible with current and co-developmental CB garments 

Target Improvements over the M40/42, MCU-2/P, and M45, 

- Reduced weight and bulk 

- Lower breathing resistance 

- Improved equipment compatibility 

- Improved and increased periods of protection 
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Joint Block 1 Glove Upgrade Program (JB2GU) 


Description, 

The JB2GU is a system that will provide protection against chemical/biological (CB) warfare 
agents. It is a component of the JSLIST ensemble and will improve upon the JB1GU effort by 
offering greater durability that will satisfy a broader set of known requirements, specifically, 
JSLIST ground and shipboard use and aviation. JB2GU will be used with the JSLIST and JPACE 
ensemble and chemical protective mask. 

Target Capabilities, 

The JB2GU will provide hand protection from liquid, vapor, and aerosol Chemical/Biological 
(CB) hazards. It will provide enhanced tactility, dexterity, and comfort over existing systems and 
can be worn in all climates. The glove will offer 24 hours of protection in a contaminated 
environment and is durable up to 30 days. 

How JB2GU is Used, 

The warfighter will use the JB2GU glove along with the JSLIST and JPACE ensembles, footwear 
and masks to create a state-of-the-art chemical/biological protective ensemble that provides 
complete percutaneous and respiratory protection against chemical and biological warfare (CBW) 
agents. 

User, 

U.S. Navy, U.S. Air Force, U.S. Army, and U.S. Marine Corps. 
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Joint Services Lightweight Standoff Chemical Agent Detector 
(JSLSCAD) 


Description, 

The Joint Services Lightweight Standoff Chemical Agent Detector (JSLSCAD) is a state-of-the- 
art detection system designed to provide U.S. Forces with enhanced capability in detecting 
chemical warfare agents. It is a lightweight, passive, and fully automatic detection system that 
scans the surrounding atmosphere for chemical warfare agent vapors. It furnishes on-the-move, 
360° coverage from a variety of tactical and reconnaissance platforms at distances up to 5 
kilometers. It is a second-generation system that significantly improves on the capabilities of the 
currently fielded M21 Remote Sensing Chemical Agent Alarm (RSCAAL). The JSLSCAD will 
provide war fighters with enhanced early warning to avoid chemically contaminated battlespace. 
When avoidance is not possible, The JSLSCAD will give personnel extra time to don MOPP 
gear. 

Mission, 

Provide standoff detection and warning for nerve, blister, and blood agent vapor clouds. 
Capabilities, 

- Detects, identifies and reports nerve, blister, and blood agent vapor clouds 

- Mounts on ground, air and sea platforms 

- Provides 360° by 60° on-the-move coverage 

- Provides up to 5 km detection range 

- No operator required 

- Automatic warning and reporting through JWARN 
Users, 

U.S. Army, U.S. Marine Corps, Future, U.S. Air Force, U.S. Navy 
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JSFDS Joint Services Family of Decontamination Systems 
[Includes JSTDS-SS] 


Description, Joint Service Family of Decontamination Systems (JSFDS) is designated to 
incorporate across the DoD a family of decontamination systems that will provide personnel, 
equipment and area decontamination of all chemical and biological warfare agents to a level safe 
for return to unrestricted operational use. 

Mission, To develop chemical and biological decontamination/application systems for 
equipment, wounded and non-wounded personnel. 

User, U.S. Army, U.S. Navy, U.S. Air Force, and U.S. Marine Corps 
M100NSN, 4230-01-466-9095 Bracket NSN, 5340-01-466-5928 
Capabilities, 

• Decon of fixed facilities, ports of entry, airfields 

• Dispensing of the entire family of decontaminants regardless of form 

• Personnel skin decontamination capability for use on casualties - with and without 
open wound. 

• Improvements over current capabilities 

• Provides a common capability to decontaminate facilities, structures, etc. 

• Introduction of skin decontamination treatment 
Unification of dispensing methods 

NOTE, Requirements for the JSTDS-SS are found in the Joint Sendee Family of 

Decontamination Systems (JSFDS) Joint Sendee Transportable Decontamination System 
(JSTDS) Milestone (MS) B ORD, Acquisition Category (ACAT) III, Increment I dated 19 
February 2004, approved 1 March 04. ( Ref p. 1, JSTDS-SS TEMP) 
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Light Vehicle Obscuration Smoke System (LVOSS) 


Description, 

The LVOSS is a self-defense smoke and riot control device externally mounted on the host 
vehicle. It consists of grenades, M7 Dischargers to fire the grenades, and installation kits to 
mount the dischargers on vehicles. 

Mission, 

Provide smoke protection and riot control for light vehicles. The LVOSS uses M90 smoke 
grenades to counter threat weapon systems operating in the visual and near-infrared portion of the 
electromagnetic spectrum. The system can also be used to deliver anti-riot grenades (L96A1, 
L97A1, M98, and M99). 

Capabilities, 

- Standardized discharger design for mounting on any family of light vehicle 

- M3 04 Installation Kit installs one M7 Dischargers on M966 HMMWVs 

- M305 Installation Kit installs four M7 Dischargers on Ml 025/M 1026 HMMWVs 

- M310 Installation Kit installs four M7 Dischargers on Ml 114 HMMWVs 

- 600 coverage per discharger 

Improvements over previous systems, 

- Provides smoke protection to light vehicles (This capability previously did not exist for light 
vehicles) 

- Uses grenades that are less toxic & are safer than those used on armored vehicles 

Users, 

U.S. Army 
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M45 Mask 


Description, 

The M45 Mask is used by all Amy aircrew members except AH-64 (Apache) helicopter pilots in 
the conduct of aviation missions anywhere in a CB environment. The mask consists of close- 
fitting eye lenses, front and side voicemitters for face-to-face and telephone communication, a 
low profile canister interoperability hose assembly to allow both hose and face mounted 
configurations, a rubber facepiece with an in-turned peripheral seal, a second skin, and a hood. 
The mask provides the required CB protection without the aid of forced ventilation air. The M45 
Mask supports the Land Warrior program and serves as the mask for personnel who cannot be 
fitted with the standard M40A1, M42A2, or MCU-2A/P protective masks. 

Mission, 

Provide aircrew and hard-to-fit personnel with protection against all known chemical and 
biological threat agents and radiological particulates. 

User, 

U.S. Amy, U.S. Navy, U.S. Air Force, and U.S. Marine Corps. 

Target Capabilities, 

Compatible with night vision devices and aircraft sighting systems 

- Microphone pass-through for aircraft communication 

- Interchangeable nosecups 

- Drink tube for liquid nutrients 

Improvements over the M24 and M43 Type II Aircraft Masks, 

- Eliminates blower and battery requirements 

- Remains flexible at extreme temperatures 

- Greater facial seal contact area 

- Removable hose can be installed on either side of the mask 
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M56 Motorized Smoke Obscurant System "The Coyote" 


Description, 

The M56 Smoke Generating System is the U.S. Army’s first visual and infrared (IR) large area 
smoke generating system. The system consists of visual, infrared, and power modules mounted 
on the back of an Ml 113 High Mobility Multipurpose Wheeled Vehicle. Fog oil is pumped into 
the exhaust gas of a turbine engine to produce visual obscuration and graphite pellets are 
pulverized to 5 micron particles and disseminated through a separate ejector to produce infrared 
obscuration. 

Mission, 

Defeat all threat reconnaissance, surveillance, and target acquisition (RSTA) systems to deny the 
enemy information, protect forces and dominate the maneuver battle. 

Capabilities, 

- Simultaneous visual/IR obscuration 

- Visual - 90 minutes (fog oil) 

- IR - 30 minutes (graphite) 

- AN/VAS-5A(V)2 Driver's Vision Enhancer 

- C-130 Transportable, Air Droppable, CH-47 Sling-Load 

- SINCGARS radio and AN/VIC-3 intercom 

Improvements over legacy systems, 

- Greater mobility 

- IR capability 

- Ease of operation 

Users, 

U.S. Army 
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Discharger, Grenade, Smoke, Countermeasure, M6 


Description, 

The M6 is a 4-tube smoke grenade discharger that enables combat vehicles to conceal themselves 
from threat surveillance, target acquisition, and weapon guidance systems. It is a 2 X 2 tube 
design weighing 9.8 lbs. 

Mission, 

Provides smoke protection to armored vehicles. 

Capabilities, 

- Four tubes are independently addressable 

- Mounts onto all armored platforms 

- Capable of firing all standard 66mm smoke grenades 
-2x2 design 

- Tubes set 14.5 degrees apart; 8"x7"x9" long; 9.8 pounds 

- Cast aluminum 

Improvements Over Previous Grenades, 

- Individual fire control capability 

- Two loaded salvos 

Users, 

U.S. Army 
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Shipboard Collective Protection System Backlit Program 


[This Information is for a more recent Shipboard CPS system, as no digital data sheet was found 
on the older (evaluated) CPS version for ships] 

Purpose, 

The shipboard Collective Protection System (CPS) Backfit Program was created to provide 
additional collective protection capabilities to existing amphibious class ships allowing personnel 
to perform mission critical operations is a Chemical, Biological, and Radiological (CBR) 
environment. 

Major Components, 

M98 Gas-particulate filter sets, pre-filter bags, filter housings, van-axial fans, fan rooms, air 
locks, pressure gages, zone pressure relief values, zone alarms and control panels and a 
decontamination station. 

Mission, 

The program installs additional CPS toxic free zones in critical areas including medical, 
command and control, rest and relief, and casualty collecting areas. Personnel working in these 
protected areas do not have to wear individual protective equipment (IPE) during or after a CBR 
attack. 

Capabilities, 

♦ Additional toxic free zones that allow ships 
personnel to operate in a CBR environment 

♦ Full-time operation 

♦ 3+ year filter life 

♦ LHA/LHD class ships now capable of 
receiving and treating contaminated casualties 
in a collectively protected environment 

User, 

U.S. Navy 
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